Metadata-Version: 2.5
Name: snapir
Version: 0.1.1
Summary: Toolkit for temporal information retrieval datasets
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: beautifulsoup4>=4.15.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: langchain-openai>=1.7.2
Requires-Dist: matplotlib>=3.11.2
Requires-Dist: pandas>=3.1.0
Requires-Dist: pillow>=13.0.0
Requires-Dist: playwright-stealth>=2.0.3
Requires-Dist: playwright>=1.64.1
Requires-Dist: protego>=0.7.0
Requires-Dist: pymupdf>=1.28.0
Requires-Dist: pypdf[crypto]>=6.21.2
Requires-Dist: serpapi>=1.1.2
Requires-Dist: tqdm>=4.70.1
Requires-Dist: trafilatura>=2.3.1
Requires-Dist: typer>=0.26.7
Provides-Extra: gemini
Requires-Dist: langchain-google-genai; extra == 'gemini'
Provides-Extra: prefect
Requires-Dist: prefect>=3.4; extra == 'prefect'
Provides-Extra: tests
Requires-Dist: bandit[toml]; extra == 'tests'
Requires-Dist: mypy; extra == 'tests'
Requires-Dist: pytest; extra == 'tests'
Requires-Dist: pytest-cov; extra == 'tests'
Requires-Dist: ruff>=0.17.3; extra == 'tests'
Description-Content-Type: text/markdown

# SnapIR
`snapir` is a python toolkit to (repeatedly) create snapshots from web search settings. This includes topic and query creation, acquisition of rankings, web pages and (synthetic) relevance judgments, ultimately building the foundation for dynamic test collections with in time relevance judgments.  

## Getting Started
```bash
pip install git+https://github.com/irgroup/snapir.git
```


### Optional LLM generation dependency

`topics generate`, `queries generate`, and `judgments generate` require `topic-gen`, which is not distributed on PyPI. Install it separately in the same Python environment as SnapIR:

```bash
pip install "topic-gen @ git+https://github.com/irgroup/topic-gen.git"
```

For a uv-managed project, use `uv add "topic-gen @ git+https://github.com/irgroup/topic-gen.git"`. Other commands and CLI help work without this dependency.

## Commands

```
SnapIR
│
├── topics
│   ├── fetch-debates       Collect source debates from Eurotopics
│   ├── generate            Generate topics from debates
│   └── select              Add a topic and its queries to a curated set
│
├── queries
│   └── generate            Generate queries for topics
│
├── serps
│   └── fetch               Fetch search results for queries
│
├── documents
│   ├── scrape              Capture documents referenced by SERPs
│   ├── extract-text        Extract missing text from saved captures (incl. PDFs)
│   ├── review              Interactively inspect screenshots
│   └── clear               Remove flagged captures so the next scrape retries them
│
├── judgments
│   ├── generate            Generate LLM relevance judgments
│   ├── create-questions    Generate the SoSci question pool
│   └── questionnaire       Build a questionnaire, screenshots, and manifest
│
└── export
    ├── rankings            Export rankings as a table
    ├── answers             Export human annotations
    ├── qrels               Aggregate human assessments into TREC qrels
    ├── snapshot            Assemble a distributable temporal collection
    └── verify              Check queries → SERPs → captures → text
```

## Configuration

Set via environment variables:

| Variable | Used by |
|---|---|
| `SNAPIR_LLM_MODEL`, `SNAPIR_LLM_BASE_URL`, `SNAPIR_LLM_API_KEY` | `topics generate`, `queries generate`, `judgments generate` (OpenAI-compatible endpoint; `gemini*` models use Google GenAI, install with `uv sync --extra gemini`) |
| `SNAPIR_LLM_TEMPERATURE` (default 1), `SNAPIR_LLM_CONCURRENCY` (default 1) | as above |
| `SERPAPI_API_KEY` | `serps fetch` |
| `BROWSER_PROFILE_DIR` (default `data/.browser_profile`), `EXTENSION_DIR` (default `data/extensions/idcac`) | `documents scrape` |
| `SNAPIR_ROBOTS_USER_AGENT` (default `SnapIR`) — robots.txt group to obey; disable the check with `--ignore-robots` | `documents scrape` |
| `SOSCI_EXPORT_URL` | default for `export answers --sosci` |

## Scheduled runs

`workflows/` has example Prefect flows that grow the topic pool and take a
daily snapshot, together with a `prefect.yaml` and a Docker setup with a
server and worker. See [workflows/README.md](workflows/README.md).


## Example
```bash
uv run snapir topics fetch-debates --output data/debates

uv run snapir topics generate --debates data/debates/debates.json --output data/topics.jsonl --prompt data/topic-emerging-v5.yaml 

uv run snapir queries generate --debates data/debates/debates.json --topics data/topics.jsonl --output data/queries.jsonl --prompt data/queries-v5.yaml

uv run snapir topics select --queries data/queries.jsonl --topics data/topics.jsonl --output-topics data/topics-selected.jsonl --output-queries data/queries-selected.jsonl --topic-id 8719997c0e34fc72dd343b0ac7ea04dc

uv run snapir serps fetch --queries data/queries-selected.jsonl --query-id 4f17e83eaac64e08cce86d6db356c5a5 --output data/serps

uv run snapir documents scrape --serps data/serps/2026-10-26_queries-selected --output data/docs

uv run snapir documents extract-text --documents data/docs/2026-10-26

uv run snapir documents review --documents data/docs/2026-10-26 --serps data/serps/2026-10-26_queries-selected 

uv run snapir documents clear --documents data/docs/2026-10-26 --dry-run

uv run snapir export rankings --serps data/serps --output data/exports/rankings.csv

uv run snapir judgments generate --serps data/serps/2026-10-26_queries-selected --documents data/docs/2026-10-26 --topics data/topics-selected.jsonl --output data/judgments/llm

uv run snapir judgments create-questions --output data/sosci

uv run snapir judgments questionnaire --serps data/serps/2026-10-26_queries-selected --documents data/docs/2026-10-26 --topics data/topics-selected.jsonl --query-id 4f17e83eaac64e08cce86d6db356c5a5 --template data/sosci/questions-relevance.xml --template data/sosci/questions-rationale.xml --template data/sosci/questions-attention.xml --questionnaire data/sosci/questionaire.xml --train data/sosci/train.xml --output data/questionnaires

uv run snapir export answers --manifest data/questionnaires/T1-IN1 --output data/judgments/answers.csv

uv run snapir export qrels --answers data/judgments/answers-clean.csv --output data/judgments/qrels.txt

uv run snapir export snapshot --serps data/serps --documents data/docs --topics data/topics-selected.jsonl --queries data/queries-selected.jsonl --qrels data/judgments/qrels.txt --output data/exports/snapshot

uv run snapir export verify --queries data/queries-selected.jsonl --serps data/serps/2026-10-26_queries-selected --documents data/docs
```


