CLI: threatlens

Inspect what the extraction engine makes of one report, without touching a database. src/threatlens/cli.py

Purpose

The threatlens command runs a single CTI report through the extraction engine and shows the result. It prints a short summary to the terminal and, with -o, writes the full result as JSON. The JSON holds every regex candidate with its offsets and context excerpt, the verifier's verdict, reason and evidence quote for it, the document metadata and, with the LLM engine, the report's scope assessment.

It exists to make the extraction stage visible on its own: to see why a value was confirmed or rejected, to compare the two verification engines on the same report, or to check a change to the extractor before ingesting anything. The JSON it writes is also the input format of the quality checker's own command line, so the next stage can be inspected the same way (see Chaining into the quality checker).

How it works

1. Parse arguments

   main() builds the parser with build_parser() and calls run(args). Expected failures (OSError, UnicodeError, ValueError, RuntimeError, OllamaError) are caught and printed as error: … on stderr with exit code 1, rather than a traceback.

   cli.py:59, cli.py:188

2. Check the source file

   run() stops straight away with source does not exist: … (exit 1) if the path is not a file.

   cli.py:121

3. Choose the engine

   --engine python calls ioc_extract_python.analyse_document() directly; no settings are loaded and no network is used.

   For the default llm engine, every LLM setting comes from threatlens.conf (or the file named by --config): the Ollama URL, the models and the [llm] performance tuning, with built-in defaults for anything missing. --ignore-calibration swaps in the built-in [llm] tuning only. A bad value stops the run with error: ….

   It then runs the setup check and prints a warning: … line on stderr for each step not yet done (no file, unedited template, not calibrated; the calibration warning is dropped with --ignore-calibration). Warnings never stop the run.

   cli.py:126–137

4. Extract and verify

   ioc_extract_llm_fs.analyse_document() reads the report (PDF via PyMuPDF, or plain text), extracts regex candidates and metadata, profiles the document once with the profiler model, then sends the candidates to the verifier model in batches. The python engine shares the same reading and candidate extraction and differs only in the verifier. See the Extraction Engine page for what happens inside.

   Both engines return the same IoCExtractionResult object, held in memory.

   cli.py:138

5. Print the summary

   Four lines: metadata (source, publish date, author count), candidate count with engine and model, verdict counts, and, for the llm engine, the primary scope with its tactical/operational/strategic flags.

   cli.py:152–173

6. Write JSON (optional)

   With -o, the result's to_dict() is written as indented UTF-8 JSON, creating parent directories as needed. --accepted-only trims the indicators list here.

   cli.py:175–183

Using it

Prerequisites

• The project installed into a virtual environment (pip install -e . from a checkout, or a plain pip install of the package), which provides the threatlens, threatlens-ingest, threatlens-calibrate and threatlens-dashboard commands. python -m threatlens.cli works too.
• For the llm engine, a reachable Ollama server with the verifier model pulled. The python engine needs neither.
• Settings are read from threatlens.conf, wherever the layout puts it (the repo root in a checkout, the per-user config folder when pip-installed, or THREATLENS_CONF), whatever the current directory. Run threatlens-calibrate once to tune its [llm] section for your Ollama server.

Synopsis

threatlens SOURCE [--engine {llm,python}] [--config FILE]
                  [--ignore-calibration] [--accepted-only] [-o FILE] [-v|-vv|-vvv]

Arguments

Where the LLM settings come from

One file, threatlens.conf; built-in defaults (ThreatLensConfig / LLMSettings in config.py) fill anything it leaves out:

threatlens-calibrate benchmarks the server and writes [ollama], the verifier model, [llm] and a [calibration] record of what it measured, keeping every other section. The dashboard's Configuration page edits the URL and models and keeps [llm].

The summary line model=… shows which verifier model was actually used.

This replaces the earlier separate llm_calibration.conf: its [llm] keys now live in threatlens.conf, and its model and URL are the [models] verifier and [ollama] url values, so there is one file and one lookup. --calibration-config became --config.

Setting up: threatlens init

pip cannot ask questions while it installs, so this is the first command to run afterwards (also threatlens-init). It asks for each setting, with the current threatlens.conf values as the defaults when the file exists, so re-running it to change one thing is quick:

1. Data folder for databases and reports (default by layout); checked for write access.
2. LLM engine? Answering no skips the Ollama questions; the python engine needs none.
3. Ollama URL, tested live. If unreachable: retry, keep it anyway, or skip LLM setup.
4. Verifier and profiler models, chosen from the models the server has pulled (or typed). The default is the configured model, else qwen3.5:9b if pulled, else the first one listed.
5. Calibrate now? Offered when the chosen model is available; runs threatlens-calibrate's benchmark at the end.
6. Database file name, and whether to create or upgrade it now.
7. Dashboard address: this machine only (127.0.0.1, default) or the local network (0.0.0.0, after a warning, since the dashboard has no login), and the port.
8. Quality weights, only if asked; re-asked until they sum to 1.00.

A summary is shown and nothing is written until it is confirmed. Then the wizard keeps the previous file as threatlens.conf.bak, writes the new one with config.save_config() (as the dashboard and calibration do; [llm] and [calibration] are kept unless it calibrates), creates or upgrades the database with the shipped migrations, fills its reference tables only if they are empty (so it is safe on a database in use), calibrates if chosen, and lists anything still to do. A file that cannot be read can be replaced from the defaults, with the old one kept.

threatlens init
threatlens init --non-interactive --ollama-url http://host:11434 --verifier qwen3.5:9b
threatlens init --non-interactive --data-dir /srv/threatlens --host 0.0.0.0 --port 8080 --no-database

--non-interactive asks nothing: it starts from the current file (or the defaults), applies --data-dir, --ollama-url, --verifier, --profiler, --database, --host, --port, and creates the database unless --no-database; --calibrate also benchmarks. Without it, threatlens init refuses to run when input is not a terminal.

First run: threatlens (llm engine), threatlens-ingest and threatlens-dashboard offer to run the wizard when there is no threatlens.conf and they are running in a terminal. Answering no continues on the built-in defaults. Jobs started by the dashboard, scripts and pipes are never prompted, and THREATLENS_NO_SETUP=1 turns the offer off.

Setup check

config.setup_status() reports how far threatlens.conf is from set up. threatlens (llm engine), threatlens-ingest and the dashboard's Ingest worker ([WARN] log lines), threatlens-dashboard at startup (SETUP: lines) and the dashboard's Configuration and Ingest pages all warn from it. Nothing is blocked; built-in defaults fill the gaps meanwhile.

threatlens.conf.example is generated from config.render(TEMPLATE, template=True), and a test fails if the two drift apart.

Where files live

threatlens/paths.py resolves every location, so the same commands work from a repo checkout and from a pip install. The layout is detected automatically:

On Windows, config and data share one folder, %LOCALAPPDATA%\threatlens (e.g. C:\Users\you\AppData\Local\threatlens, holding threatlens.conf, db\ and reports\); on macOS both are ~/Library/Application Support/threatlens. Overrides, highest first: THREATLENS_CONF for the config file; THREATLENS_HOME, then [paths] data_dir in threatlens.conf, for the data folder (which then holds db/ and reports/). Report paths stored in the database are relative to the data folder, so a checkout's existing docs/database/CTI/… rows still resolve. The dashboard's Configuration page shows the folders in use.

Calibrating the LLM settings: threatlens-calibrate

Also python -m threatlens.extractors.calibration. It warms the model, times structured verification requests at several batch sizes, picks the smallest batch within 8% of the best throughput, labels the hardware (cpu-minimal … gpu-high) and suggests [llm] values. Saved, it writes [ollama] url, [models] verifier, [llm] and [calibration] into threatlens.conf and keeps every other section. An existing file that cannot be read is reported before any benchmarking and is never overwritten.

threatlens-calibrate                                   # re-tune the configured model and server
threatlens-calibrate --model qwen3.5:9b --ollama-url http://host:11434 --save

For a setup script, calibrate() and save_calibration() in extractors/calibration.py do the same without the command line. pip cannot run calibration while installing (it never prompts and needs a running Ollama), so it is a step after installation.

Examples

# Summary only, default llm engine
threatlens docs/database/CTI/report.pdf

# No LLM available: heuristic verifier
threatlens report.pdf --engine python

# Full result as JSON, with batch progress
threatlens report.pdf -o out/report.llm.json -v

# Only accepted candidates in the JSON
threatlens report.pdf -o out/report.accepted.json --accepted-only

# One-off settings file, or built-in [llm] tuning
threatlens report.pdf --config ~/threatlens.low_cpu.conf
threatlens report.pdf --ignore-calibration

Console output

From one of the evaluation corpus reports with the python engine:

$ threatlens docs/database/CTI/fdf53fac0af9cced76179fb1b9df0ce009952e36.pdf --engine python -o py.json
metadata: Skia/PDF m90, published 2022-02-21, 0 author(s)
extract:  3 candidates from fdf53fac0af9cced76179fb1b9df0ce009952e36.pdf (python engine, model=python-heuristics)
verify:   3 confirmed, 0 context, 0 rejected
output:   py.json

The llm engine adds a fourth line, for example scope: tactical (tactical=True, operational=False, strategic=False). The verify line does not count uncertain verdicts; the JSON summary does.

JSON output

One indicators entry, abridged:

{
  "id": "ba27359faf1a32066466",
  "category": "network",
  "indicator_type": "url",
  "value": "blog.alyac.co.kr/4501",
  "normalized_value": "blog.alyac.co.kr/4501",
  "start": 119,
  "end": 140,
  "context": "… customer center! blog.alyac.co.kr/4501 Detailed content body …",
  "method": "regex:schemeless-url",
  "attributes": { "was_defanged": false, "document_section_hint": "…" },
  "verification": {
    "candidate_id": "ba27359faf1a32066466",
    "status": "confirmed_ioc",
    "confidence": 0.76,
    "reason": "Candidate appears in malicious CTI context but lacks stronger structural evidence.",
    "evidence": "nearby_label=ESRC | …",
    "corrected_category": "network",
    "corrected_indicator_type": "url"
  },
  "accepted": true
}

start/end are character offsets into the extracted text, and context is the window either side of the match that the verifier saw. verification.confidence is confidence in the verdict, not in maliciousness. accepted is true for confirmed_ioc and context_entity.

Chaining into the quality checker

The quality checker reads these JSON files directly, so the QA stage can be inspected without a database too:

threatlens report.pdf -o llm.json
threatlens report.pdf --engine python -o python.json

python -m threatlens.quality.quality_checker llm.json --show all
python -m threatlens.quality.quality_checker llm.json --compare python.json

--compare enables the cross-verifier agreement grading. Both files must come from the same report.

Exit codes and errors