LLM Analysis Deviation Report: ex2, ex3, ex4

This report compares three complete extraction runs over the same 103-PDF corpus, looking at each report in turn: db/ex2_threatlens.db, db/ex3_threatlens.db and db/ex4_threatlens.db. For every report it asks how far the stored results differ between runs, and turns that into a single deviation value that is then averaged per report.

Two other databases were left out: ex_threatlens.db is an earlier run covering only 100 of the 103 reports, and breakit_ex2_threatlens.db is a test copy of ex2. ex3 and ex4 come from the current pipeline (llm engine, qwen3.5:9b, temperature: 0). ex2 was ingested between 22 Sep and 7 Oct 2026 under an earlier pipeline build; its sources do not record an extraction engine.

A report entry is one indicator sighted in one report, i.e. one row of the sightings table. Entries are matched across runs by report filename plus the indicator's (ioc_type, normalised) key. For each entry and each pair of runs, five components are scored from 0 (identical) to 1 (completely different):

An entry that appears in only one run would score 1 on every component. None did. 2 reports have no stored sightings in any run (6b8d1b3f…, f14bf397…), so they are shown as n/a and left out of the averages, which therefore cover 101 reports.

Quality score is kept out of the LLM deviation on purpose. It is computed downstream from the scoring weights and from the date each run was scored (timeliness), not from the LLM. Folding it in would count configuration and calendar effects as LLM variance, so it gets its own section.

Weights as recorded in each indicator's score_components.weights, which are the weights actually applied. All three databases now use the defaults, which also match the scoring_config table in each. ex3 and ex4 were originally scored with test weights left in threatlens.conf (0.10 / 0.70 / 0.20 / 0.00). They were rescored with the defaults on 13 Oct, with timeliness evaluated as of each run's original scoring time, so the comparison below is like for like. The same day, all three had first_seen / last_seen backfilled from report publish dates (timeliness reads last_seen) and were rescored again on the same basis.

The stored score is Σ(weight × dimension) / 5 × 100, with each dimension on a 0–5 scale.

Only 97 of 6,239 entries score differently between ex2 and ex3/ex4, by 0.08 points on average. Two dimensions account for all of it:

• Timeliness differs on 63 entries (52 indicators). It depends on when a run was scored: reports published just before ex2 was scored on 7 Oct had moved into an older age band by the time ex3/ex4 were scored.
• Completeness differs on 34 entries (7 indicators), following the context-field changes in the LLM deviation above.

Accuracy and relevance are identical in all three runs. The two dimension counts are not additive: an entry can differ in both, and a change in one dimension can round to the same score.

Ranked by the three-way index. Because ex3 = ex4, this ranking is the ex2 ↔ ex3 ranking.

• The current pipeline is reproducible. ex3 and ex4 were run a day apart and match on every content field. That fits temperature: 0 inference on a fixed model. The per-report candidate, confirmed, context and rejected counts in the ex3 and ex4 logs also agree for all 103 reports. Repeat runs of this build therefore cannot measure LLM run-to-run variance; any such measurement needs a non-zero temperature or a different model.
• ex2's deviation is small, and it comes from changes between pipeline builds, not run-to-run noise. Candidate extraction is unchanged: the indicator sets, sightings and excerpts are identical. The differences are in LLM verdicts, confidences, a few context fields, quality flags, and one defanged value (cdn[.]discordapp[.]com), which ex3/ex4 store refanged. ex2 was built under an earlier build of the extractor and quality checker, so these differences are expected.
• Quality scores are comparable across all three databases. ex3 and ex4 were rescored with ex2's weights on 13 Oct. The small remaining difference comes from timeliness (scoring date) and from completeness, which follows the context fields above.

Generated from the three SQLite databases on 13 Oct 2026; quality-score figures updated the same day after ex3 and ex4 were rescored with the default weights, and all three databases had first_seen / last_seen backfilled from report publish dates and were rescored as of their original scoring times. Timestamps (ingested_at, scored_at, seen_at, raised_at) and database row IDs are excluded from the comparison because they always differ between runs.