Metadata-Version: 2.4
Name: scm-ai-extract
Version: 0.1.0
Summary: Three-stage MinerU and LLM pipeline for extracting upstream supply-chain-management sustainability practice rows from corporate reports.
License: MIT
Keywords: esg,extraction,llm,openrouter,pdf,supply-chain,sustainability
Requires-Python: <3.14,>=3.11
Requires-Dist: openai>=1.40
Requires-Dist: pandas>=2.0
Requires-Dist: pydantic>=2.7
Requires-Dist: pymupdf>=1.24
Requires-Dist: python-dotenv>=1.0
Requires-Dist: pywebview>=5.0
Requires-Dist: requests>=2.31
Requires-Dist: tenacity>=8.2
Requires-Dist: tiktoken>=0.7
Requires-Dist: tomlkit>=0.13
Requires-Dist: tqdm>=4.66
Description-Content-Type: text/markdown

# scm-ai-extract

A command-line and GUI tool that extracts **upstream supply-chain-management
(SCM) sustainability practice/keyword rows** from corporate sustainability/ESG
report PDFs and writes one tidy CSV per report.

The production pipeline is now three stages:

```text
PDF
  -> Stage 1 parse   (MinerU precision API batch parsing)
  -> Stage 2 compact (cheap text model: paragraph relevance + summaries)
  -> Stage 3 extract (indexed relevant paragraphs -> final CSV)
```

Stage 1 calls MinerU's precision API and preserves raw paragraph evidence under `output/.markdown/...`.
Stage 2 never repeats relevant paragraph text in model output; it only records
paragraph ids, relevance, and summaries for irrelevant paragraphs. Stage 3 sends
`README.md` plus indexed relevant paragraphs and maps returned ids back to the
raw evidence text before writing CSV.

## Quick Start

```bash
# One OpenRouter key powers Stage 2 and Stage 3.
export OPENROUTER_API_KEY=sk-or-...

# MinerU precision API token for Stage 1.
export MINERU_API_KEY=...

# From an installed package:
uvx scm-ai-extract --input input --output output

# From this checkout:
uv run scm-ai-extract --input input --output output

# Pass keys directly instead of using environment variables:
uvx scm-ai-extract --input input --output output \
  --openrouter-api-key 'YOUR_OPENROUTER_KEY' \
  --mineru-api-key 'YOUR_MINERU_KEY'
```

If `config.toml` is missing, the CLI asks whether to create it from the packaged
template. Before paid API calls, the terminal prints a cost estimate and asks
for confirmation. In GUI mode, that confirmation is shown in the window.

Stage 1 uses MinerU's precision batch API with `model_version = "vlm"`. Reports
over 200 pages are split into <=200-page PDFs before upload.
For faster Stage 1 uploads, pass `--cdn-token "$(cat .local-secrets/scm_cdn_token.txt)"`.
That route uploads PDFs to the managed Cloudflare/R2 Worker first, submits
short-lived signed URLs to MinerU, and keeps checksum-addressed objects in R2
for up to 7 days so repeat runs can reuse identical PDFs without re-uploading.
The R2 bucket lifecycle rule deletes those temporary objects automatically.

## Configuration

`config.toml` is the single source of truth.

| Section | What it controls |
| --- | --- |
| `[openrouter]` | API base URL and environment variable name for the key. |
| `[stage1]` | MinerU precision API endpoint/env var, model version, parser flags, split size, batch size, polling, and timeout. |
| `[stage2]` | Paragraph compaction model, batch limits, and compaction/README prompts. |
| `[stage3]` | Final extraction model, batch limits, extraction prompt, and time rules. |
| `[output]` | `format = [{ name, description }, ...]`; descriptions define categorical guidance. |

Default models:

| Stage | Default |
| --- | --- |
| Stage 2 compact | `google/gemini-2.5-flash-lite` |
| Stage 3 extract | `google/gemini-3.1-flash-lite` |

Stage 1 is not an OpenRouter model; OpenRouter cost estimates cover Stage 2 and Stage 3 only.

## CLI

```text
scm-ai-extract [--config PATH] [--input PATH] [--output DIR]
               [--stage {all,parse,compact,extract}]
               [--resume] [--dry-run] [--yes]
               [--max-concurrency N] [--max-pdf-concurrency N]
               [--stage-2-model SLUG] [--stage-3-model SLUG]
               [--temperature F] [--top-p F]
               [--reasoning-effort {none,minimal,low,medium,high,xhigh}]
               [--max-output-tokens N]
               [--openrouter-api-key KEY] [--mineru-api-key KEY]
               [--cdn-token TOKEN] [--gui] [--verbose]
```

| Flag | Default | Meaning |
| --- | --- | --- |
| `--input` | required outside GUI | A PDF file or a folder of PDFs, searched recursively. |
| `--output` | `output` | Output folder; CSVs mirror input subfolders and artifacts go under `output/.markdown/`. |
| `--stage` | `all` | `parse`, `compact`, or `extract` can run from cached upstream artifacts. |
| `--resume` | off | Reuse complete artifacts; otherwise existing outputs are ignored and rebuilt. |
| `--dry-run` | off | Discover inputs and estimate Stage 2/3 API cost without calling models. |
| `--yes` | off | Skip cost confirmation. |
| `--max-concurrency` | `32` | Shared Stage 2/3 LLM worker pool size. |
| `--max-pdf-concurrency` | `8` | Parallel report orchestration for folder runs. |
| `--stage-2-model` | config value | Override compaction model for this run. |
| `--stage-3-model` | config value | Override final extraction model for this run. |
| `--openrouter-api-key` | env var named by `[openrouter].api_key_env` | OpenRouter key for Stage 2/3. |
| `--mineru-api-key` | env var named by `[stage1].api_key_env` | MinerU precision API key for Stage 1. |
| `--cdn-token` | off | Use the managed Cloudflare/R2 upload path for MinerU Stage 1. |

Generation overrides (`--temperature`, `--top-p`, `--reasoning-effort`,
`--max-output-tokens`) apply to Stage 3 only.

Examples:

```bash
# A tree of PDFs; output mirrors input subfolders.
uv run scm-ai-extract --input reports/ --output output/

# Estimate cost only.
uv run scm-ai-extract --input reports/ --dry-run

# Run MinerU parse now, then compact/extract later.
uv run scm-ai-extract --input reports/ --stage parse
uv run scm-ai-extract --input reports/ --stage compact --resume
uv run scm-ai-extract --input reports/ --stage extract --resume

# Override models for a single run.
uv run scm-ai-extract --input reports/ \
  --stage-2-model google/gemini-2.5-flash-lite \
  --stage-3-model deepseek/deepseek-v4-flash --yes

# Open the config/progress GUI.
uv run scm-ai-extract --gui
```

## Artifacts

For an input PDF `reports/nested/A.pdf` and `--output output`, artifacts are
written to:

```text
output/
  nested/A_final_dataset.csv
  .progress/
    nested/A_final_dataset.csv.done
  .markdown/
    nested/
      A.pdf/
        mineru/
        paragraphs.raw.jsonl
        paragraphs.compact.jsonl
        README.md
```

Resume checks are stage-specific:

| Stage | Complete when |
| --- | --- |
| Stage 1 | MinerU content list exists and `paragraphs.raw.jsonl` validates. |
| Stage 2 | `paragraphs.compact.jsonl` covers every raw paragraph id and `README.md` exists. |
| Stage 3 | CSV has expected columns and the hidden `.progress/...csv.done` marker validates. |

Folder runs continue after true per-report failures and mark failed reports in
the summary.

## Benchmarking Stage 3 Models

`scm-ai-extract-benchmark` compares final extraction models using completed
parse+compact artifacts. It does not run MinerU or Stage 2 compaction.

```bash
uv run scm-ai-extract-benchmark --markdown-dir output/.markdown --dry-run

uv run scm-ai-extract-benchmark \
  --markdown-dir output/.markdown \
  --reports NASDAQ_AAPL_2021 NASDAQ_NVDA_2024 NYSE_F_2021 \
  --models deepseek/deepseek-v4-flash deepseek/deepseek-v4-pro \
  --budget-usd 1.00 --yes
```

Results are written to `output/benchmark_stage3/summary.csv` and
`output/benchmark_stage3/summary_by_model.csv`.

## GUI

`--gui` opens a pywebview window that edits `config.toml`, computes the run
review/cost estimate, and then shows progress in-process. Progress rows show
Stage 1, Stage 2, and Stage 3 bars for each report.

For local GUI development:

```bash
pnpm -C frontend install
pnpm -C frontend dev
SCM_AI_EXTRACT_GUI_DEV_URL=http://localhost:5173 uv run scm-ai-extract --gui
```

Build distributable GUI assets with:

```bash
pnpm -C frontend build
```

## Development

```bash
uv sync
uv run ruff check src tests
uv run python -m unittest discover -s tests -v
pnpm -C frontend build
git diff --check
```

## Output Columns

The final CSV columns are:

```text
source_report, page, paragraph, keyword, reasoning, scm_category,
scm_subcategory, topic, time_orientation, report_pages, scm_structure
```

`paragraph` is always filled from `paragraphs.raw.jsonl`, not copied from the
Stage 3 model response.

## Layout

| Path | What |
| --- | --- |
| `src/scm_ai_extract/` | Installable package: CLI, pipeline, stages, GUI backend. |
| `src/scm_ai_extract/resources/config.toml` | Packaged default config template. |
| `frontend/` | React/Vite source for the GUI window. |
| `eval/` | Gold evidence, mention CSVs, review notes, and evaluation material. |
| `small_input/`, `input/` | Sample/report PDFs used during development. |
