Metadata-Version: 2.4
Name: llmexer
Version: 0.4.1
Summary: Framework and CLI utility to create and curate datasets (e.g., publications, metadata, LLM prompts, etc.) and orchestrate (design, run, evaluate) various LLM experiments on them
Author: vdmitriyev, ellariel
Maintainer: vdmitriyev
License-Expression: MIT
Project-URL: homepage, https://github.com/vdmitriyev/llmexer
Project-URL: changelog, https://github.com/vdmitriyev/llmexer/blob/main/HISTORY.rst
Project-URL: repository, https://github.com/vdmitriyev/llmexer.git
Project-URL: documentation, https://vdmitriyev.github.io/llmexer/
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Utilities
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.9.0
Requires-Dist: rich>=13.7.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: Jinja2>=3.1.6
Requires-Dist: typing-extensions>=4.0.0
Requires-Dist: pandas>=3.0.1
Requires-Dist: requests>=2.33.0
Requires-Dist: pypdf>=6.10.2
Requires-Dist: langdetect>=1.0.9
Requires-Dist: pydantic>=2.12.5
Requires-Dist: openai>=2.30.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: SQLAlchemy>=2.0.36
Requires-Dist: nbformat>=5.10.4
Dynamic: license-file

## 🔰About

![PyPI Version](https://img.shields.io/pypi/v/llmexer?style=flat)
![PyPI License](https://img.shields.io/pypi/l/llmexer?style=flat)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/llmexer?style=flat)

`llmexer` is a framework and CLI utility to create and curate datasets (e.g., publications, metadata, LLM prompts, etc.) and orchestrate (design, run, evaluate) various LLM experiments on them

> 🪄 The philosophy of the tool is: `everything` is a `file`. Projects, experiments, searches, configs, and further items will be saved as files. The CLI helps you to modify most of the files, but the same files could be modified manually (e.g., adding a new LLM model, modification of search search or paper as PDFs could be manually added, further more, a SQLite database with generated experiments could be inspected and edited etc.).

## ✨ Early `beta` warning

The package is still in its early beta stage, so breaking changes may be introduced at short notice.

## 📦 Installation

* Install using `pip`
    ```
    pip install --upgrade llmexer
    ```
* Install using `uv`
    ```
    uv pip install --upgrade llmexer
    ```
* Check `configuration` section afterward

## ⚙️ Configuration

This tool requires access to local or remote running LLMs. It uses a `.env` file to securely load your API credentials, and also set the current project.

1.  Create a `.env` file in the root of the project
2.  Set the current project ID (optional):
    ```env
    PROJECT_ID=20260330-3a9adf70
    ```
3.  Configure the docling backend (optional, used by `papers extract --processor docling`):
    ```env
    DOCLING_URL=http://localhost:5001/
    DOCLING_USER=myuser
    DOCLING_PASSWORD=mypassword
    ```
4.  Set the API key for LLM providers (optional, used by `experiment run` for ollama, vllm, litellm, OpenAI, Gemini):
    ```env
    # Base URL for a specific provider
    PROVIDER_OLLAMA_URL=http://localhost:11434/v1
    PROVIDER_VLLM_URL=http://localhost:8000/v1

    # The API key for a specific provider
    PROVIDER_OPENAI_KEY=sk-...
    ```
     The pattern is `PROVIDER_<PROVIDER_UPPER>_URL` and `PROVIDER_<PROVIDER_UPPER>_KEY` where `<PROVIDER_UPPER>` is the provider name in uppercase (e.g. `OLLAMA`, `VLLM`, `LITELLM`, `OPENAI`, `GEMINI`).

     The `litellm` provider talks to a [LiteLLM](https://docs.litellm.ai/) proxy sitting in front of a vLLM backend. Unlike the local providers it has **no default URL** and **requires an API token** - both must be set, otherwise `experiment run` aborts immediately with a clear error instead of failing later with an opaque `401`:
    ```env
    PROVIDER_LITELLM_URL=https://your-litellm-proxy.example.org/v1
    PROVIDER_LITELLM_KEY=sk-...
    ```
5.  Enable the OpenAlex search engine (optional, used by `search run` as a second source after Semantic Scholar; skipped if unset):
    ```env
    OPENALEX_API_KEY=...
    # Optional: cap on OpenAlex results processed per query (default 5000)
    MAX_OPEN_ALEX_RESPONSES=5000
    # Optional: used for DOI downloads via Unpaywall and also as the OpenAlex polite-pool mailto
    UNPAYWALL_EMAIL=you@example.com
    ```
6. If you would like to change the envs based on the project run (e.g., just test a LLM provider for a particular project, via stats of a search, etc.), you could also pass a custom file as `.env` to the CLI:
    ```
    llmexer --env-file custom.env
    ```
7. By default CLI will work with current directory to save files, but it could be redefined
    ```
    LLMEXER_BASEDIR=my-projects
    ```

## Documentation

[Documentation](https://vdmitriyev.github.io/llmexer/)

## 🚀 Getting Started

A typical end-to-end workflow for collecting and processing papers inside a project:

**1. Create a new project**
```bash
llmexer project create
# Output: created project '20260402-a1b2c3d4'
```

**2. Give it a meaningful name**
```bash
llmexer project rename --old-id 20260402-a1b2c3d4 --new-id llm-survey-2026
```

**3. Initialise the project structure**

Scaffold a standard `experiment/` subfolder with template CSVs and a prompt file:
```bash
llmexer experiment init --pid llm-survey-2026
```

<details markdown="1">

Initialization of the project creates following files (inside <PROJECT_NAME> folder):

| File | Description |
| :-------- | :---------- |
| `experiment/llms-for-experiment.csv` | List of models to use (`provider`, `model_name`, `profile_name`, `notes`); pre-filled with `gemma4:31b`, `phi4:14b`. Each row is one model run under one `llm-params.csv` profile — list a model twice to run it under two profiles. |
| `experiment/data.csv` | Input data rows (`ID`, `Title`, `Abstract`). |
| `experiment/mapping.csv` | Maps data IDs to prompt IDs; pre-filled with `D01;prompt01` and `D02;prompt01`. |
| `experiment/prompts/prompt01.txt` | A starter Jinja2 prompt template using `{{title}}` and `{{abstract}}`. |
| `experiment/llm-params.csv` | LLM hyperparameter profiles. Identity columns: `provider`, `model_name`, `profile_name`; universal columns: `temperature`, `top_p`, `max_tokens`; provider-grouped columns: `ollama_context_window`, `ollama_repeat_penalty` (ollama), `vllm_min_p`, `vllm_best_of` (vllm), `openai_seed` (openai), `gemini_thinking_level` (gemini), `litellm_min_p`, `litellm_best_of` (litellm). Pre-filled with example profiles for `ollama`, `openai`, `vllm`, `gemini`, and `litellm`. |

</details>

**4. Generate the full experiment database**

After filling in `experiment/llms-for-experiment.csv`, `experiment/data.csv`, `experiment/mapping.csv`, `experiment/llm-params.csv`, and your Jinja2 prompt templates and use `map` to bring all together. The two CSVs are matched on `provider`, `model_name` and `profile_name`; that combination must be unique in each file, or `generate` reports it and stops.

<details>
* `experiment/mapping.csv` can be built for you rather than written by hand — handy after
`experiment copy-papers` or `experiment copy-search` has replaced `data.csv` with `P01`/`S01`-style IDs.
* `map` pairs every data row with every selected prompt and backs up any existing `mapping.csv`
</details>

```bash
llmexer experiment map --pid llm-survey-2026 --prompt prompt01,prompt02
```

At the end, generate database file with experiments:
```bash
llmexer experiment generate --pid llm-survey-2026
```

<details markdown="1">

This renders every (data row × prompt × LLM models × LLM parameters) combination and writes a self-contained SQLite database `experiment/experiment_<YYYYMMDD>_<NN>.db` (`<NN>` is a zero-padded counter starting at `01`). Each LLM provider gets **two** tables. `experiment_<provider>` (e.g. `experiment_ollama`) holds the rendered prompt, model identity and the SHA-256 hashes — plus the result columns that `experiment run` fills in later. `params_<provider>` (e.g. `params_ollama`) holds one row per hyperparameter set, so a profile is stored once instead of being repeated on every row of the cross join. The two are joined on `params_code` (built as `MODELNAME_PROVIDER`) together with `profile_name`. The `code` field encodes each combination as `DATAID_PROMPTID_MODELNAME_PROFILENAME`.

Profiles are matched to models on **both** `provider` and `model_name` (stripped, case-sensitive), so a profile only applies to the provider it was written for. A model with no matching profile is reported and skipped. Use `--dry-run` to preview the row count without writing:
```bash
llmexer --dry-run experiment generate --pid llm-survey-2026
```

</details>

> 💡 **Hint — external tools:** the generated experiment store is a SQLite database
> (`experiment/experiment_*.db`). Beyond the CLI, you can open and edit it directly with any
> external SQLite tool — for example [DBeaver](https://dbeaver.io/). To see each row next to the
> hyperparameters it was run with, join the two tables:
>
> ```sql
> SELECT e.*, p.* FROM experiment_ollama e
> JOIN params_ollama p
>   ON e.params_code = p.params_code AND e.profile_name = p.profile_name;
> ```

**5. Extend an existing experiment database (optional)**

Added a model, a profile or a few mapping rows after generating? `update` appends the missing combinations to an existing database instead of starting a new one:
```bash
llmexer experiment update --pid llm-survey-2026
```

<details markdown="1">

The input CSVs are re-read and cross-joined exactly as `generate` does, then compared with the database: combinations it does not hold yet are appended after the highest existing `ID`, and everything already stored — including the results `run` has collected — is left untouched. With no `--file` the newest `experiment_*.db` is updated; `--dry-run` reports the row count without writing.

Hyperparameters are never rewritten. If a profile of `llm-params.csv` still carries its old name but different values, `update` prints the differing columns (`db=` vs `csv=`) and stops without writing anything — otherwise rows generated before and after the change would be indistinguishable. Give the changed parameters a new `profile_name`, point the matching `llms-for-experiment.csv` row at it, and run `update` again. An edited prompt template or `data.csv` cell is reported the same way, as a warning; the stored rows keep the text they were generated with.

</details>

**6. Try a single experiment combination (optional)**

Before running the whole cross join, check what one prompt actually returns for one data row under one profile:
```bash
llmexer experiment try --pid llm-survey-2026 \
  --prompt prompt01 --profile ollama-default --data-id D01
```

<details markdown="1">

`try` is `generate` and `run` for a single combination. The three params are validated first (if prompt is known, `--data-id` exists and  profile available), otherwise a call is aborted before any LLM call. The model and provider must come from the profile's row in `llm-params.csv` (when one profile name covers several models, `try` lists the candidates and asks for `--model`/ `--provider`).

Every try is appended to `try_experiment_<provider>` and `try_param_<provider>` in the experiment database (`--file`, newest by default), so a series of tries is kept as a history and each one records the parameters it actually ran with. The generated experiment rows, their results and `experiment stats` are untouched. Option `--dry-run` prints the rendered prompt without calling anything; it is accepted both as the global flag (`llmexer --dry-run experiment try ...`) and as an option of `try` itself (`llmexer experiment try ... --dry-run`).

</details>

**7. Run the experiment - call LLMs and collect results**

Once `experiment generate` has produced the database, run all combinations:
```bash
llmexer experiment run --pid llm-survey-2026 --file experiment_<SAMPLE>.db
```

Every option of `experiment run` — previewing a run, restricting it to one provider, model or profile, running a single combination, and running several calls at once — is covered in [Scenario 4](#-scenario-4-advanced-usage-of-the-experiment-cli-commands).

**8. Inspect experiment statistics**

Get aggregate statistics (total, finished, running, errors, total tokens, and per-provider / per-model breakdowns). The per-model table reports, for each model, the `provider` serving it, `requests`, `finished`, `open` (pending/unrun), `time total` (HH:MM:SS elapsed over finished requests), `average time` (HH:MM:SS mean elapsed per finished request), and `tokens` (summed over finished requests). With no `--file` it reads the project's single `experiment_*.db` (pass `--file` if several exist):
```bash
llmexer experiment stats --pid llm-survey-2026
```
Pass `--file` to inspect a specific database instead:
```bash
llmexer experiment stats --pid llm-survey-2026 --file experiment_<SAMPLE>.db
```

**9. Export the results as HTML (optional)**

Render the finished experiment as a single, self-contained page for screening or sharing:
```bash
llmexer experiment export --pid llm-survey-2026 --file experiment_<SAMPLE>.db
```

<details markdown="1">

Writes `experiment_<SAMPLE>.html` next to the database: sortable columns, per-column filters,
row counters, dark mode and a copy button on every cell. Ten columns are exported —
`provider`, `model`, `profile`, `code`, `response_text`, `tokens`, `status`, `seconds`,
`timestamp`, `try`.

The `try` column holds a ready-to-run command that re-runs that one combination — copy the
cell and paste it into a shell:
```bash
llmexer experiment try --pid llm-survey-2026 --file experiment_<SAMPLE>.db \
  --data-id D01 --prompt prompt01 --profile ollama-default \
  --model llama3.3:latest --provider ollama
```
</details>

**10. Analyse the results in a notebook (optional)**

Scaffold a Jupyter workspace for the project and open it:
```bash
llmexer analysis init --pid llm-survey-2026
```

<details markdown="1">

Creates `analysis/` next to `experiment/`, `papers/` and `searches/`, containing
`analyse_experiment.ipynb`, `analyse_searches.ipynb` and the `transform.py` / `stats.py` /
`plots.py` modules they call. Open `analysis/analyse_experiment.ipynb` and run all cells: it
loads the experiment database named in `DB_FILE`, parses each model answer from JSON into its
own columns, writes just those answers to a CSV next to the notebook, counts the distinct values
of every answer column and charts the yes/no and countable ones, then prints the statistics and
renders the rest of the charts inline. `analyse_searches.ipynb` does the same for one search — set
`SEARCH_INDEX` to pick which.

The modules are copies, so a project can grow its own analysis — `%autoreload` picks up every
edit without a kernel restart. The notebooks import only those copies, never `llmexer`, so the
folder is self-contained. `llmexer analysis init --rewrite` puts the shipped versions back,
moving your copies into `analysis/.backup/` first. Running the notebooks needs the optional packages:
`uv pip install -e . --group analysis`.

</details>

The API key is read from `.env` (pattern -> `PROVIDER_<PROVIDER_UPPER>_KEY`).

P.S.: CLI interfaces could become very complex with the time, thus refer to the `--help` to get options and parameters of the utility:
```bash
llmexer --help
```

## 📢 Scenario 1: Gathering data for projects by adding papers

**1. Add papers to the project** - local file

From a local file:
```bash
llmexer papers add --pid llm-survey-2026 --file ~/Downloads/attention-is-all-you-need.pdf
```
**2. Add papers to the project** - from directory

From a directory of PDFs:
```bash
llmexer papers add --pid llm-survey-2026 --directory ~/Downloads/papers/
```

**3. Add papers to the project** - from url
From a URL:
```bash
llmexer papers add --pid llm-survey-2026 --url https://arxiv.org/pdf/1706.03762
```

## 📢 Scenario 2: Gathering data for projects by running search

**1. Run a literature search**

Create a search configuration and run it:
```bash
llmexer search create --pid llm-survey-2026 --query "large language models"
llmexer search list --pid llm-survey-2026
llmexer search run --pid llm-survey-2026 --file 20260401-bfdd863d.yaml
```

Or run directly from a query string:
```bash
llmexer search run --pid llm-survey-2026 --query "large language models" --limit 500
```

Results are saved as `<ID>__results.csv` in `searches/` (the raw JSON is saved as `<ID>__results_raw.json` in `searches/jsons/`).

**2. Filter search results by excluding rows (optional)**

`filter` **excludes** rows by one or more criteria and writes `<ID>__filtered.csv`. Filters chain: each run reads the existing `__filtered.csv` (or the `__results.csv` if none) and rewrites it. Combine `--language`, `--source`, `--doi`, `--downloaded` in one run:
```bash
# drop German rows and rows still not downloaded
llmexer search filter --pid llm-survey-2026 --file 20260401-bfdd863d.yaml --language de --downloaded
```
`--file` is optional — omit it to apply the same filters to **every** search in the project. Every applied filter (per search) is recorded in `searches/logs/filters-applied.log`.

**3. Export search results as HTML (optional)**

```bash
llmexer search export --pid llm-survey-2026 --file 20260401-bfdd863d.yaml
```
Writes `<ID>__results.html` next to the CSV: sortable columns, per-column filters, row counters, dark mode, clickable DOIs, copy buttons, collapsible abstracts. `--csv-file <name>.csv` exports a single CSV; omit both to export every search.

## 📢 Scenario 3: Gathering data by downloading papers and extracting text

**1. Download open-access papers by DOI via Unpaywall**

Download by DOI (one or more):
```bash
llmexer papers download --pid llm-survey-2026 --doi 10.1038/nature12373 --email you@example.com
```
Download from a full search result CSV (downloads all papers with a DOI, names each file `YEAR_AUTHOR_TITLE_DOI.pdf`):
```bash
llmexer papers download --pid llm-survey-2026 --search-file 20260401-bfdd863d__results.csv
```
Or from a filtered CSV to download only the papers that passed the language filter:
```bash
llmexer papers download --pid llm-survey-2026 --search-file 20260401-bfdd863d__filtered.csv
```
Failed downloads are saved automatically as `20260401-bfdd863d__results_download_failed.csv` (columns: `doi`, `url`, `title`, `desired_filename`, `downloaded`) in the `searches/logs/` folder. After a `--search-file` download completes, the search is automatically synced against the `papers/` folder in **existing-only** mode — it updates `pdf_downloaded` (and text/markdown companions) for the listed rows but does not add new rows for unrelated PDFs.

**2. Extract text from all added papers** - pypdf

Using the default `pypdf` backend (saves `.txt` files):
```bash
llmexer papers extract --pid llm-survey-2026
```

**3. Extract text from all added papers** - docling

Using the `docling` backend for richer Markdown output (saves `.md` files), reading connection details from `.env`:
```bash
llmexer papers extract --pid llm-survey-2026 --processor docling
```

Override `.env` connection settings at runtime:
```bash
llmexer papers extract --pid llm-survey-2026 --processor docling \
  --docling-url http://myserver:5001/ \
  --docling-user admin \
  --docling-password secret
```

By default, papers that already have an extracted file are skipped. Use `--rewrite` to force re-extraction:
```bash
llmexer papers extract --pid llm-survey-2026 --rewrite
```

## 📢 Scenario 4: Advanced usage of the experiment CLI commands

<details>
<summary>Details about how does the `experiment run` CLI command works</summary>
`experiment run` reads every row from the generated database (`experiment_*.db`, joining each row with its provider's `params_<provider>` table) and calls the appropriate LLM. With no `--file` it uses the newest `experiment_*.db`. Results are written **back into the same database in place** — each row's response, status, token usage, and timestamps are updated on its provider table, so the database stays the single source of truth (no separate results file). Re-running skips rows that already finished successfully and updates the rest. Each individual call is also saved as a JSON file under `experiment/responses/`. Both the per-call JSON and the database `response_json` column include the **complete raw backend response** under `raw_response` (all provider fields — e.g. `finish_reason`, per-token `usage`, and ollama extras like `eval_count` / `*_duration`), not just the response text and total token count.
</details>

**1. Preview a run without calling any LLM**

Use `--dry-run` to see the row count first:
```bash
llmexer --dry-run experiment run --pid llm-survey-2026
```

**2. Run only a specific provider's rows by applying filter**

Useful when only one backend is available (e.g. a local ollama):
```bash
llmexer experiment run --pid llm-survey-2026 --filter-provider ollama
```

**3. Narrow the run to one model or one hyperparameter profile**

Both match the name in full and case-sensitively:

```bash
llmexer experiment run --pid llm-survey-2026 --filter-model gemma4:31b
```

Filter by profile
```bash
llmexer experiment run --pid llm-survey-2026 --filter-profile ollama-creative
```

Combining two filter
```bash
llmexer experiment run --pid llm-survey-2026 --filter-provider ollama --filter-model gemma4:31b
```

**4. Run a single experiment out of the set of experiments**

By its `code`:
```bash
llmexer experiment run --pid llm-survey-2026 \
  --file experiment_<SAMPLE>.db --code 1
```

**5. Run several LLM calls at once**

Most of a run is spent waiting on the LLM backend, so `--parallel-calls` lets several calls be in flight at once. The N limit is global to all (never more than N calls in total, whatever mix of providers the rows use). Default is `1` (i.e. the sequential runs). Results are still written **one at a time**.
```bash
llmexer experiment run --pid llm-survey-2026 --parallel-calls 4
```

**6. Try a single experiment combination (optional)**

Before running the whole cross join, check what one prompt actually returns for one data row under one profile:
```bash
llmexer experiment try --pid llm-survey-2026 \
  --prompt prompt01 --profile ollama-default --data-id D01
```

#### Using Current Project ID

Many commands support the `--pid` parameter to specify which project to work with. If you set `PROJECT_ID` in your `.env` file, you can omit this parameter and the commands will use the current project automatically:

```bash
# Set in .env
PROJECT_ID=my-project

# These commands will use my-project automatically
llmexer search run --query "machine learning"
```

You can still override the current project by explicitly providing `--pid`:
```bash
llmexer search run --pid different-project --query "deep learning"
```

## 🗂️ CLI category: **project**

The `project` (alias: `proj`) category provides commands for managing LLM projects (the top-level container for experiments, papers, and searches):

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `create` | Create a new project folder under `.projects/` using format `YYYYMMDD-GUID`. Accepts an optional custom ID. | `llmexer project create --id my-project` |
| `current` | Display the current project ID loaded from `.env`. | `llmexer project current` |
| `rename` | Rename an existing project. Uses `PROJECT_ID` from `.env` if `--old-id` is omitted. | `llmexer project rename --old-id old-name --new-id new-name` |

## 🧪 CLI category: **experiment**

The `experiment` (alias: `exp`) category provides commands for initialising, generating, and running LLM experiments inside a project:

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `init` | Initialise an existing project with a standard folder structure (`experiment/`, `experiment/prompts/`) and template files: `llms-for-experiment.csv` (`provider;model_name;profile_name;notes`, pre-filled with example ollama models pointing at the `ollama-default` / `ollama-creative` profiles), `data.csv`, `mapping.csv` (pre-filled with D01 and D02 rows), `prompts/prompt01.txt` (Jinja2 template using `{{title}}` and `{{abstract}}`), and `llm-params.csv` (hyperparameter profiles; universal: `temperature`, `top_p`, `max_tokens`; ollama: `ollama_context_window`, `ollama_repeat_penalty`; vllm: `vllm_min_p`, `vllm_best_of`; openai: `openai_seed`; gemini: `gemini_thinking_level`). Raises an error if already initialised. | `llmexer experiment init --pid my-project` |
| `copy-papers` | Copy parsed papers (`.md`/`.txt`) from the project's `papers/` folder into `experiment/data.csv` as rows `ID;filename;content`, with IDs `P01`, `P02`, … ordered alphabetically by filename (`.md` preferred over `.txt` when both exist). An existing `data.csv` is backed up to `data_backup_<YYYYMMDD>_<NN>.csv` first. | `llmexer experiment copy-papers --pid my-project` |
| `copy-search` | Copy a search results CSV (`--file`, absolute or relative to the project's `searches/` folder) into `experiment/data.csv` as rows `ID;Title;Abstract;doi;authors`, with IDs `S01`, `S02`, … preserving the source file's row order. An existing `data.csv` is backed up to `data_backup_<YYYYMMDD>_<NN>.csv` first. | `llmexer experiment copy-search --pid my-project --file <SEARCH_ID>__results.csv` |
| `map` | Build `experiment/mapping.csv` by pairing every row of `data.csv` with the selected prompt(s) from `prompts/` — a cross join, written prompt by prompt. Use `--prompt` to pick templates (repeatable, and each value may itself be a comma-separated list; the `.txt` extension is optional); omit it to use every prompt in `prompts/`. A named prompt that does not exist aborts the command with every missing name listed, leaving `mapping.csv` untouched. An existing `mapping.csv` is backed up to `mapping_backup_<YYYYMMDD>_<NN>.csv` first. Supports `--dry-run`. | `llmexer experiment map --pid my-project --prompt prompt01,prompt02` |
| `generate` | Render all (data row × prompt × LLM models × LLM parameters) combinations and write a self-contained SQLite database `experiment/experiment_<YYYYMMDD>_<NN>.db` (`<NN>` is a zero-padded counter starting at `01`). Each LLM provider gets two tables: `experiment_<provider>` (e.g. `experiment_ollama`) with columns `ID`, `code` (`DATAID_PROMPTID_MODELNAME_PROFILENAME`), `prompt`, `tokens_estimate`, `original_data`, `model_name`, `provider_name`, the `params_code` / `profile_name` join key, the `prompt_hash` / `original_data_hash` columns and the result columns filled in by `run` (`response_text`, `status`, `state`, `call_count`, `prompt_tokens`, `completion_tokens`, `total_tokens`, `elapsed_seconds`, `timestamp`, `response_json`); and `params_<provider>` (e.g. `params_ollama`) with one row per hyperparameter set from `llm-params.csv` — primary key `(params_code, profile_name)`, then `temperature`, `top_p`, `max_tokens` plus the provider-specific ones (e.g. `ollama_context_window`, `ollama_repeat_penalty`). Profiles from `llm-params.csv` are matched on all three of `provider`, `model_name` and `profile_name`; a model with no matching profile is reported and skipped, and a duplicated combination in either file is reported in red and aborts before anything is written. Rows are sorted by model order from `llms-for-experiment.csv`. Supports `--dry-run`. | `llmexer experiment generate --pid my-project` |
| `update` | Add combinations added to the input CSVs into an already generated database, instead of generating a new one. The CSVs are re-read and cross-joined exactly as `generate` does, then compared with the database: combinations it does not hold yet are appended after the highest existing `ID`, while stored rows and the results `run` collected for them are left untouched (a provider new to the database gets its `experiment_<provider>` / `params_<provider>` pair created). A profile of `llm-params.csv` whose values changed under an unchanged `profile_name` aborts the command with the differing columns reported as `db=` vs `csv=` and nothing written — rename the profile and re-run. Stored rows whose prompt template or `data.csv` text has since changed are reported as a warning and left as they are. Supports `--dry-run` and `--file` (choose a specific `.db`, defaults to the newest). | `llmexer experiment update --pid my-project` |
| `try` | Render and run **one** `data.csv` row × prompt × profile combination and print the response — `generate` and `run` at once, for a single try. Takes `--prompt` (a template from `prompts/`, `.txt` optional), `--profile` (a profile name from `llm-params.csv`, matched in full and case-sensitively) and `--data-id` (an `ID` from `data.csv`); all three are validated up front and an unknown name aborts before any LLM call, listing what is available. The model and provider come from the profile's row; `--model` / `--provider` disambiguate a profile name shared by several models. The response is printed under a `model` / `provider` / prompt, completion and total token header, saved as JSON under `experiment/responses/`, and appended to the `try_experiment_<provider>` / `try_param_<provider>` tables of the database (`--file`, newest by default) — one row per try in each, so every try keeps the parameters it ran with. Generated rows and `stats` are unaffected. On a real try the rendered prompt is printed first, then a `Waiting for LLM provider to answer ...` spinner while the call is in flight. Supports `--dry-run` both as the global flag and as its own option (`-d`), which renders and prints the prompt and calls nothing. | `llmexer experiment try --pid my-project --prompt prompt01 --profile ollama-default --data-id D01 --dry-run` |
| `run` | Execute every row in the generated database `experiment_*.db` (no separate params file needed — all columns are embedded). Calls each LLM via the OpenAI SDK (supports ollama, vllm, litellm, openai, gemini) and writes results **back into the same database in place** (response, status, `prompt_tokens` / `completion_tokens` / `total_tokens`, timestamps, plus the complete raw backend response under `raw_response`); re-runs skip rows that already finished successfully and update the rest. Individual JSON responses are saved under `experiment/responses/`. Supports `--dry-run`, `--file` (choose a specific `.db`, defaults to the newest), `--filter-provider` (only run rows for a specific provider, case-insensitive), `--filter-model` / `--filter-profile` (only run rows whose `model_name` / `profile_name` matches in full, case-sensitively), `--code` (run a single combination by its `code`), `--parallel-calls` (how many LLM calls may be in flight at once across all providers, default `1`; results are still written one at a time). The filters combine with AND; when none of the filtered rows exist the command reports it and exits cleanly. API key read from the `PROVIDER_<PROVIDER_UPPER>_KEY` env var; URL from `PROVIDER_<PROVIDER_UPPER>_URL` or built-in defaults (`litellm` requires both to be set explicitly). Requires `openai` package (`pip install openai`). | `llmexer experiment run --pid my-project --filter-provider ollama` |
| `stats` | Show aggregate statistics from a project's experiment database: totals (total, finished, running, errors), total tokens, and per-provider / per-model breakdowns rendered as Rich tables. The Models table has columns `Model`, `Provider`, `requests`, `finished`, `open` (pending/unrun), `time total` (HH:MM:SS over finished requests), `average time` (HH:MM:SS mean per finished request), and `tokens` (summed over finished requests); it is grouped by (model, provider) and sorted by model then provider, so the same model served by two providers is reported once per provider rather than merged into one row. With no `--file` it reads the project's single `experiment_*.db` (pass `--file` to choose one when several exist). | `llmexer experiment stats --pid my-project` |
| `list` | List all projects with their initialization state and generated experiment databases, with optional sorting by name or date. | `llmexer experiment list --sort-by date --desc` |
| `export` | Render a generated experiment database as an HTML page next to it (same name, `.html`). Exports `provider`, `model`, `profile`, `code`, `response_text`, `tokens`, `status`, `seconds`, `timestamp` and `try` for every row, run or not. Sortable columns, per-column filters, row counters, dark mode and a copy button on every cell. `code` is collapsed to 5 characters behind a `more` toggle; `response_text` is pretty-printed as JSON when it parses (a ```json fence is stripped first) and kept as plain text when it does not; `tokens` is `total_tokens`; `seconds` is rounded to one decimal and `timestamp` trimmed to whole seconds; `try` is a ready-to-run `experiment try` command that re-runs that single combination. `--file` chooses a database (newest by default), `--rewrite` overwrites; respects `--dry-run`. | `llmexer experiment export --pid my-project` |

## 📊 CLI category: **analysis**

The `analysis` (aliases: `analyse`, `analyze`) category scaffolds a ready-to-run Jupyter workspace for analysing a project. It creates `analysis/` next to `experiment/`, `papers/` and `searches/`, holding two notebooks and the Python modules they call. The notebooks contain no logic — the cells configure paths, call functions and display the result — so the analysis itself stays testable code. The modules are **copies** and the notebooks import only them, never `llmexer`, so a scaffolded `analysis/` folder is self-contained and can be tweaked per project.

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `init` | Create `<project>/analysis/` with `analyse_experiment.ipynb` (loads the experiment database, flattens each `response_text` from JSON into columns, exports **only those parsed answers** as a CSV, then profiles them — how many distinct values each answer column took and whether it is yes/no, a countable rating or a small set of labels, with a bar chart per chartable column — before printing the summary, per-model responses, response times, token usage and failures, and rendering four more charts) and `analyse_searches.ipynb` (loads **one** search plus the `papers/` inventory, then breaks it down by search engine, year, open access and language). Alongside them it copies `transform.py`, `stats.py` and `plots.py` — edit them freely, the notebooks pick the changes up via `%autoreload`. What each notebook reads is set by literals written in at scaffold time and editable by hand: `DB_FILE` (the newest `experiment_*.db`) and `SEARCHES` (one entry per search, naming its YAML and the `results` / `filter` CSVs written from it) with `SEARCH_INDEX` choosing which one to analyse — searches are never combined, since two queries are two different populations. Neither literal updates itself: after another `experiment generate` or `search run`, edit the value or re-scaffold. Every notebook is validated with `nbformat` before it is written. Existing files are **never** overwritten without `--rewrite`, which first copies each one to `analysis/.backup/<stem>_backup_<YYYYMMDD>_<NN><ext>`, so the analysis folder keeps showing only the files meant to be opened. Works on a project with no experiment or searches yet — the notebooks report an empty project rather than failing. Respects `--dry-run`. | `llmexer analysis init --pid my-project` |

Running the notebooks needs the optional analysis packages: `uv pip install -e . --group analysis`.

## 📑 CLI category: **papers**

The `papers` category provides commands for managing PDF papers within a project:

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `add --file` | Copy a single PDF into the project's `papers/` folder. | `llmexer papers add --file /path/to/paper.pdf` |
| `add --directory` | Recursively copy all PDFs from a directory. Already-existing papers are skipped. | `llmexer papers add --directory /path/to/folder` |
| `add --url` | Download a PDF from a URL into the project's `papers/` folder. | `llmexer papers add --url https://example.com/paper.pdf` |
| `download --doi` | Download one or more open-access PDFs by DOI using the Unpaywall API. Email required via `--email` or `UNPAYWALL_EMAIL` env var. | `llmexer papers download --doi 10.1038/nature12373 --email you@example.com` |
| `download --search-file` | Download all papers from a search result CSV (inside `searches/`), including filtered CSVs (`__filtered.csv`). Files are named `YEAR_AUTHOR_TITLE_DOI.pdf`. On completion, auto-runs `search sync` to reconcile the search against `papers/` (updates `pdf_downloaded`, txt/markdown). Failures saved as `<stem>_download_failed.csv` in `searches/logs/`. | `llmexer papers download --search-file 20260401-abc123__filtered.csv` |
| `extract` | Extract text from all PDFs in `papers/`. Default `pypdf` backend saves `.txt`; `docling` backend sends PDFs to a remote docling-serve instance and saves `.md`. Connection details (`DOCLING_URL`, `DOCLING_USER`, `DOCLING_PASSWORD`) read from `.env`; overridable via `--docling-url`, `--docling-user`, `--docling-password`. Already-extracted files are skipped unless `--rewrite` is passed. Pass `--skip-if-md` (pypdf only) to skip PDFs that already have a `.md` extraction. Reports counts for `Extracted`, `Skipped`, `Existing`, and `Error`. | `llmexer papers extract --pid my-project --processor docling` |

## 🔍 CLI category: **search**

The `search` category provides commands for managing and running literature searches. `search run` queries the Semantic Scholar bulk API and, when `OPENALEX_API_KEY` is set, additionally queries the OpenAlex Works API as a second engine — keeping only publications not already found by Semantic Scholar (matched by DOI, falling back to title). OpenAlex is skipped if the key is unset. OpenAlex rows are marked `entry_source="OpenAlex"`.

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `create` | Create a search configuration YAML file in the project's `searches/` folder. | `llmexer search create --query "machine learning"` |
| `list` | List all search YAML configs in the project's `searches/` folder as a table (columns: `#`, `Name`, `Query`, `Year`, `Created`, `Results`). Prints a next-step hint referencing the latest search file. | `llmexer search list --pid my-project` |
| `rename` | Rename a search ID and all its associated files (`<id>.yaml`, `<id>__results.csv`, `<id>__filtered.csv`, `<id>__results_raw.json` under `searches/jsons/`, and `<id>__results_download_failed.csv` under `searches/logs/`). Accepts a full `.yaml` filename for `--old-id`. | `llmexer search rename --old-id 20260401-abc123 --new-id my-search` |
| `run --query` | Run a search directly from a query string. Saves `<ID>__results.csv` to `searches/` and `<ID>__results_raw.json` to `searches/jsons/`. CSV columns include: `sem_scholar_paper_id`, `year`, `title`, `authors`, `abstract`, `isOpenAccess`, `doi`, `language`, `referenceCount`, `citationCount`, `entry_source`, `pdf_filename`, `txt_filename`, `markdown_filename`, `pdf_downloaded`. Raw JSON also contains `fieldsOfStudy`, `citationStyles`, `publicationTypes`. | `llmexer search run --query "neural networks" --limit 200` |
| `run --file` | Run a search loading parameters from an existing YAML config. Use `--rewrite` to overwrite existing result files. | `llmexer search run --file 20260401-abc123.yaml` |
| `stats` | Display statistics for a completed search: papers per year and a stats breakdown (open access, language, downloaded, entry source, txt/markdown presence), stacked for results and filtered CSVs. Without `--file`, falls back to the merged file(s) (`<pid>__merged_results.csv` / `<pid>__merged_filtered.csv`) if present. | `llmexer search stats --file 20260401-abc123.yaml` |
| `filter` | **Exclude** rows from a search and rewrite `<ID>__filtered.csv`. Reads the existing `__filtered.csv` if present (filters chain), else `__results.csv`. `--file` is optional — omit it to filter every search in the project. Combinable criteria, each applied in order and logged: `--language <code>` / `--source <value>` / `--doi <value>` drop rows equal to the value; `--downloaded` drops rows not yet downloaded. Each applied filter appends a line to `searches/logs/filters-applied.log`. | `llmexer search filter --file 20260401-abc123.yaml --language de --downloaded` |
| `merge` | Merge the project's search CSVs into two deduplicated files: `<pid>__merged_results.csv` (from `*__results.csv`) and `<pid>__merged_filtered.csv` (from `*__filtered.csv`). Deduplicates by DOI (falling back to title); adds a `0/1` column per search (named after its YAML id) and a `duplicates_counter` column (number of duplicate occurrences, i.e. searches found in minus one). Rows are sorted by year (newest first; blank years last). Use `--rewrite` to overwrite; respects `--dry-run`. | `llmexer search merge --pid my-project` |
| `sync` | Reconcile `<ID>__results.csv` (and `<ID>__filtered.csv` if present) against the project's `papers/` folder. Updates `pdf_downloaded`, `txt_filename`, and `markdown_filename` for existing rows. By default only files listed in existing rows are updated; pass `--add-local-extra-pdfs` to also append new rows for PDFs in `papers/` not yet listed (marked `entry_source="manually added"`). `--file` is optional: with it a single search is synced, without it every search in the project is synced (in that case `--add-local-extra-pdfs` is not applied). Respects `--dry-run`. | `llmexer search sync --file 20260401-abc123.yaml` |
| `export` | Render sanitized search result CSVs as HTML pages next to the CSV (same name, `.html`). Sortable columns, per-column filters, row counters, dark mode, clickable DOIs, copy buttons, `more`/`less` toggles. `--file` takes a search ID/YAML name, `--csv-file` a CSV filename inside `searches/`; omit both to export the whole project. `--rewrite` overwrites; respects `--dry-run`. | `llmexer search export --file 20260401-abc123.yaml` |

Semantic Scholar API Documentation: [Paper bulk search](https://api.semanticscholar.org/api-docs/#tag/Paper-Data/operation/get_graph_paper_bulk_search) -> this can be used to formulate more sophisticated query string

OpenAlex API Documentation: [Works](https://docs.openalex.org/api-entities/works) -> queried as a second engine when `OPENALEX_API_KEY` is set

## 🔎 CLI category: **self**

The `self` category provides introspection commands for the llmexer CLI itself:

| Command   | Description | Command Example |
|-----------|-------------|-----------------|
| `version` | Print the current llmexer package version. | `llmexer self version` |
| `envs` | Display all llmexer-relevant environment variables as a table. `PROJECT_ID` is highlighted in bold cyan; `DOCLING_PASSWORD` is masked as `********` when set. | `llmexer self envs` |


## 💡 Additional: Renaming PDFs with `pdf-renamer` tool

Before adding papers to a project, you can automatically rename them by their bibliographic metadata (year, journal, authors, title) using the external [`pdf-renamer`](https://github.com/MicheleCotrufo/pdf-renamer) tool.

No installation is needed — run it directly with `uvx`.

Rename using custom format: year - authors (et al.) - title:
```bash
uvx --from pdf-renamer pdfrenamer -f "{YYYY}_{A3etal}_{T}" /path/to/pdfs
```
Rename recursively (include subdirectories)
```bash
uvx --from pdf-renamer pdfrenamer /path/to/pdfs -sf
```

## 💡 Additional: Extract BiBTeX

There is also a possibility to extract BiBTeX of a publication as follows
```bash
uvx --from pdf2bib pdf2bib -s bibtex.bib /path/to/pdfs
```

## 💡 Additional: Migrating a pre-`0.4.0` experiment database

`0.4.0` replaced the single `usage_tokens` result column with `prompt_tokens` / `completion_tokens` / `total_tokens`, the two new columns placed immediately before `total_tokens`. A database written by an earlier version is now rejected when it is opened, so it either has to be regenerated with `experiment generate` or converted in place.

`scripts/migrate_usage_tokens.py` does the conversion. It is a standalone stdlib-only script, not a CLI command: it takes the `.db` file itself, rebuilds each `experiment_<provider>` / `try_experiment_<provider>` table so the two new columns land in the right position, reads the prompt/completion split out of the stored per-call JSON (`response_json`), carries `usage_tokens` over into `total_tokens` where that was never written, and rewrites the stored payloads to the new keys. A row whose provider never reported a breakdown keeps `NULL` in the two new columns rather than a misleading `0`.

```bash
# report what would change, touch nothing
python scripts/migrate_usage_tokens.py .projects/my-project/experiment/experiment_20260101_01.db --dry-run

# convert in place; the original is copied to <stem>_backup_<YYYYMMDD>_<NN>.db first
python scripts/migrate_usage_tokens.py .projects/my-project/experiment/experiment_20260101_01.db
```

Running it twice is a no-op — an already-migrated table is reported and skipped. Pass `--no-backup` to skip the copy.

## ✏️ CLI UI

CLI feature overview:
```
llmexer --help
```

![help CLI](https://raw.githubusercontent.com/vdmitriyev/llmexer/refs/heads/main/docs/cli-ui.png)

Checking the statistics of a performed search query directly in CLI:
```
llmexer search stats --file <filename>
```
![search CLI](https://raw.githubusercontent.com/vdmitriyev/llmexer/refs/heads/main/docs/cli-ui-search-stats.png)

List existing projects directly in CLI with the current project highlighted:
```
llmexer experiment list
```
![experiment CLI](https://raw.githubusercontent.com/vdmitriyev/llmexer/refs/heads/main/docs/cli-ui-experiment-list.png)

Export the search results as a static HTML file to make it easier to screen the initial results:
```
llmexer search export
```
![experiment CLI](https://raw.githubusercontent.com/vdmitriyev/llmexer/refs/heads/main/docs/search-export.png)



## 🧩 Development Setup

This guide walks through setting up the project for local development using `uv`.

1. Create a new virtual environment in a `.venv` directory and activates it.
    ```bash
    uv venv
    ```
1. Activate the environment (macOS/Linux):
   ```
   source .venv/bin/activate
   ```
1. Activate the environment (Windows):
    ```
    call .venv/Scripts/activate.bat
    ```
1.  Install package in **editable mode** with **dev** dependencies
    Installing the package in **editable mode** (`-e`) is the key to development. It links the `llmexer` command in your environment directly to your source code.
    ```bash
    uv pip install -e . --group dev
    ```
    The `dev` group already pulls in the `analysis` group (matplotlib, seaborn, ipykernel,
    jupyterlab), so the scaffolded notebooks run and their tests import cleanly. To install
    only what is needed to *run* the notebooks:
    ```bash
    uv pip install -e . --group analysis
    ```

## License

[MIT](https://github.com/vdmitriyev/llmexer/blob/main/LICENSE)
