Dashboard / Web UI

The dashboard is a FastAPI application that serves a browser UI directly over the live SQLite database. It uses the same SQLAlchemy models the ingest pipeline writes to, so what it shows and edits is the live data, not a copy. It is where a reader reviews indicators, reports and quality flags, and also where reports are uploaded and ingested and where the per-machine settings in threatlens.conf are edited.

Running it

# with the virtual environment active; needs the dashboard extra:
#   pip install "threatlens-cti[dashboard]"     (or pip install -e ".[dashboard]" in a checkout)
threatlens-dashboard                                   # threatlens.conf [dashboard], default http://127.0.0.1:8000
threatlens-dashboard --host 0.0.0.0 --port 8000 --reload  # reachable from the local network

# or directly
uvicorn threatlens.dashboard.server:app --reload

# or in the background, from src/threatlens/dashboard/ (repo checkout only)
./start.sh [PORT]        # also start.bat / start.ps1; stop with stop.*

The address comes from threatlens.conf [dashboard] host / port (set by threatlens init), default 127.0.0.1:8000; --host / --port override it. With no threatlens.conf and a terminal, the dashboard first offers to run the setup wizard.

The start scripts run the server in the background, write its PID to .server.pid and its output to server.log, refuse to start a second copy, and use the repo's .venv when there is one. THREATLENS_DASHBOARD_HOST overrides the host. They are not part of the pip package; threatlens-dashboard is.

The pages, their JS/CSS and these documentation pages are package data, so a pip-installed copy serves the full UI and /docs/ with no repo checkout.

At startup the dashboard checks threatlens.conf (see setup check) and prints a SETUP: line for each step not yet done: the file is missing, is still the unedited template, or its [llm] settings have not been calibrated. These are warnings only; the dashboard starts on built-in defaults regardless.

Which database it uses

At startup the dashboard opens the database folder (db/ in a checkout, <data folder>/db/ when pip-installed; see where files live) + threatlens.conf [database] default (else threatlens.db). THREATLENS_DB_URL or THREATLENS_DB_FILE override that. The header's Database dropdown switches to any other .db file in that folder at runtime, without a restart. Every page, the API, the database browser and newly queued ingest jobs follow the switch.

Pages

Main page (/)

A header with links to the other pages, the database dropdown and four stat cards: indicator count, report count, open quality flags and mean quality score. Below it, three tabs:

An indicator's engine is not stored on the indicator: it is the combination of the engines of the reports it was seen in, so a value seen by both reads llm+python.

Report page (/reports/{id})

The report's metadata and quality summary, then a table of its indicators with the four dimension scores, quality score, confidence and flags. Under it is the original PDF, rendered page by page on the server with PyMuPDF, with every place an indicator appears highlighted. Rows and highlights link both ways: click a row to jump to the value in the PDF. The In PDF column counts how often each value was found; buttons step between pages with hits, and the view fits to page or width.

The matching is deliberately looser than an exact search, because the stored value often differs from the printed one (wrapped URLs and hashes, defanged forms). The PDF itself is not in the database: raw_documents.file_path points at it in docs/database/CTI/, so if the file is moved or deleted the view is unavailable but the data is not affected.

Indicator page (/indicators/{id})

Everything known about one indicator, with each number explained:

• Confidence breakdown: the verdict, the verdict confidence, the agreement grade and its multiplier, computed with the quality checker's own functions so the page uses the same arithmetic that produced the number;
• Quality score breakdown: each dimension's score, its weight and its share of the total, from score_components;
• Verifier reasoning and quality flags;
• Threat context: the structured context (actors, malware, campaigns…);
• Reports containing this indicator, with their excerpts;
• Seen alongside: other indicators that share reports with this one.

Upload Reports (/uploads)

Lists the PDFs in the report folder, the folder ingest reads from (docs/database/CTI/ in a checkout, gitignored; <data folder>/reports/ when pip-installed; created on the first upload or download), with size, date and whether each is already in the active database (matched by path, or by content hash for a renamed copy). Files can be uploaded (several at once, optionally overwriting), opened in the browser, or removed.

This page only changes files on disk. Uploading does not ingest, and removing a file does not touch the database; that is the Reports tab's delete. Uploads are accepted only if the content starts with %PDF-, are written under a temporary name and renamed when complete, and must be plain filenames (no paths).

Ingest (/ingest)

Runs the ingest pipeline (extract, quality checks, persist, rescore) on chosen reports:

• Engine: llm or python, with a status line showing which model the llm engine would use and whether the Ollama server has it. The status chip reads ready, unavailable, or ready, setup incomplete when the server is fine but threatlens.conf is missing, unedited or uncalibrated; the missing steps are listed beside it and in the chip's tooltip. The same warnings appear as [WARN] lines at the top of each job's log.
• Log detail: verbosity 0–3 (default 2), i.e. the extractor's -v level.
• Not yet ingested: the report files not in the active database; pick some and queue them.
• Download from URL: fetches a PDF into the report folder and queues it.
• Queue: each job's status (queued, running, done, failed, cancelled), progress estimate and live log; cancel one or all, clear finished ones.

How the queue works:

• Jobs run one at a time, each in its own ingest_worker.py process. Every job writes the same SQLite file, and one LLM job already keeps Ollama busy. A separate process means a slow job can be cancelled by killing it, and an extractor crash cannot take the dashboard down.
• Progress comes from the worker's machine-readable @@progress lines and, during LLM verification, its batch counter, which is where nearly all the time goes. The log is polled from the page while the job runs.
• Jobs are pinned to the database that was active when queued, so switching databases mid-run does not redirect them.
• The queue and download list live in memory: restarting the dashboard drops queued jobs and kills the running worker. The database stays consistent, because a report is only committed after its extraction and quality checks finish.

URL downloads are locked down because the dashboard may be reachable from the network: http(s) on ports 80/443 only, no credentials in the URL, no proxies, and every connection, including each redirect hop, must go to an address the server has itself resolved and checked to be publicly routable. A response is accepted only if it starts with %PDF-.

Configuration (/config)

Views and edits threatlens.conf, the same file the CLI and the ingest worker read:

A Setup incomplete box at the top lists anything the setup check found: no file yet (defaults are shown, and saving creates it), the unedited template, or no calibration. Saving from this page counts as reviewing the template, even with its values unchanged; calibration needs threatlens-calibrate.

Values are validated before saving, and saved changes apply to the next ingest without a restart. Re-score existing indicators runs rescore_from_db() on the active database. Changing weights does not change stored scores until that runs or the next ingest does.

The weights here apply to every database the dashboard or ingest touches. Test weights left in place will be applied to the next evaluation run, which is what happened to ex3 and ex4 (see the evaluation reports).

Export (/export) and the tabs' Export buttons

Downloads data from the active database (the header's Database dropdown). The Export page in the header walks through four choices, with a live count and the first rows as a preview:

Each column can be switched on or off. Every tab also has an Export button with a format selector: it downloads exactly what that tab is showing, on every page rather than just the visible one. The Indicators button uses the tab's filters and sort order, so the file matches the table row for row (the two share one query, threatlens.db.queries).

• CSV is UTF-8 with a byte-order mark, so Excel opens it correctly. Nested fields become JSON text. A value starting with =, +, - or @ is prefixed with an apostrophe: report text is untrusted, and a spreadsheet would otherwise run it as a formula.
• JSON is {"export": {…}, "items": […]}: the dataset, database, time, filters and columns, then the records, with nested fields kept as objects.
• STIX 2.1 (indicators) is a bundle that imports into MISP, OpenCTI and other platforms. It holds an identity for ThreatLens, one indicator per indicator with a STIX pattern for its type, and a report per source report that links to the indicators seen in it. Patterns use the refanged value (hxxp://evil[.]com becomes http://evil.com), because a defanged pattern matches nothing; the value as written is kept in the description. Identifiers are derived from the indicator's type and value, so exporting twice gives the same ids and an importing platform does not create duplicates. A screened indicator (confidence 0) is exported as unknown, not malicious-activity. A type with no STIX observable is left out, and the download reports how many were.

Files are named after the database, dataset and time, e.g. ex4_threatlens_indicators_20261018-0930.stix.json. The same exports are available from Python: threatlens.exporters.export(session, "indicators", "csv", filters={...}).

Database browser (/db-browser/) and documentation (/docs/)

The header's Database link opens Datasette, embedded in the dashboard, over the active database. It shows every table's raw rows and columns, unlike the curated tabs, and is read-only. It follows the database dropdown. Documentation serves these pages from src/threatlens/docs/, which ships with the package.

Editing and deleting: what changes

API

Every page is static HTML plus a JavaScript module calling a JSON API. FastAPI's interactive API docs are at /api/docs and /api/redoc.

GET /api/config includes a setup object (exists, edited, calibrated, ready, warnings), and GET /api/ingest/engines includes setup_warnings; the Configuration and Ingest pages show them.

A middleware makes browsers revalidate the pages and their JS/CSS on every load, so a code change shows up without a hard refresh.

Source

• src/threatlens/dashboard/server.py: the FastAPI app, page routes, mounts (/static, /docs, /db-browser), the threatlens-dashboard entry point
• src/threatlens/dashboard/routes/: the eight API routers above
• src/threatlens/dashboard/db.py: engine/session wiring and runtime database switching
• src/threatlens/dashboard/ingest_jobs.py, ingest_worker.py: the ingest queue and the pipeline worker (threatlens-ingest)
• src/threatlens/dashboard/url_downloads.py: URL downloads and their network safeguards
• src/threatlens/exporters/: the export datasets and the CSV, JSON and STIX 2.1 writers (stix.py), used by routes/export.py; src/threatlens/db/queries.py: the indicator filters and sorting shared with the Indicators tab
• src/threatlens/dashboard/pdf_view.py: PDF page rendering and indicator highlighting
• src/threatlens/dashboard/db_browser.py: the embedded Datasette browser
• src/threatlens/dashboard/*.html, static/js/, static/css/: the pages and their per-page scripts (package data, listed in pyproject.toml)
• src/threatlens/docs/: these documentation pages, served at /docs/ (package data)
• start.* / stop.*: background start and stop for bash, cmd and PowerShell (checkout only, not packaged)