Metadata-Version: 2.4
Name: arxiv-ingest
Version: 0.10.0
Summary: Fetch recent arXiv papers and scaffold them into a structured research wiki
Project-URL: Homepage, https://github.com/yamadan96/arxiv-ingest
Project-URL: Repository, https://github.com/yamadan96/arxiv-ingest
License: MIT
Keywords: arxiv,knowledge-management,papers,research,wiki
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: arxiv>=4.0.0
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: rich>=15.0.0
Provides-Extra: summarize
Requires-Dist: anthropic>=0.40.0; extra == 'summarize'
Description-Content-Type: text/markdown

# arxiv-ingest

[![Tests](https://github.com/yamadan96/arxiv-ingest/actions/workflows/tests.yml/badge.svg)](https://github.com/yamadan96/arxiv-ingest/actions/workflows/tests.yml)
[![PyPI version](https://img.shields.io/pypi/v/arxiv-ingest)](https://pypi.org/project/arxiv-ingest/)
[![Python 3.11+](https://img.shields.io/pypi/pyversions/arxiv-ingest)](https://pypi.org/project/arxiv-ingest/)

**Fetch recent arXiv papers by keyword and scaffold them into structured research notes.**

Works standalone as a CLI, or integrates with Claude Code for LLM-assisted evidence extraction and wiki generation.

## Features

- **Keyword-based collection** — list topics in `config.yaml`, no code needed
- **3-layer output** — `sources/` (metadata) · `evidence/` (claims) · `wiki/` (synthesis)
- **Safe re-runs** — already-filled files are never overwritten
- **GitHub Actions template** — auto-runs Mon–Fri, `workflow_dispatch` for on-demand runs
- **Any wiki** — point `output_dir` at any directory

## Quick start

```bash
# Install (requires Python 3.11+)
pip install arxiv-ingest

# Or from source:
git clone https://github.com/yamadan96/arxiv-ingest
cd arxiv-ingest
uv sync

# Configure
cp config.yaml.example config.yaml
# Edit config.yaml: set keywords, output_dir, and category mappings

# Fetch papers and generate skeletons
arxiv-ingest run                # fetch + generate in one step

# Or step by step:
arxiv-ingest fetch              # saves data/fetched.json
arxiv-ingest generate           # creates skeleton files in output_dir
```

## Configuration (`config.yaml`)

```yaml
output_dir: "../research-wiki"  # path to your wiki directory
max_results: 20                 # papers per keyword
days_back: 7                    # look back N days

keywords:
  - "Vision-Language Model transformer"
  - "LoRA PEFT fine-tuning language model"
  - "LLM reasoning chain-of-thought"

# Only accept papers in these arXiv categories:
allowed_arxiv_categories: [cs.CV, cs.CL, cs.LG, cs.AI, cs.RO, stat.ML]

# Primary category must be in this list (prevents off-topic papers):
require_primary_in: [cs.CV, cs.CL, cs.AI, cs.RO, stat.ML]

# arXiv category → wiki folder name:
category_map:
  cs.CV: "Multimodal"
  cs.CL: "Post_Training"
  cs.AI: "Reasoning"
  cs.RO: "Physical_AI"
  stat.ML: "Pretraining"
```

See `config.yaml.example` for the full reference with all options documented.

## Output format

Skeleton files are created at:

```
output_dir/
├── sources/{Category}/{slug}.md      # Metadata (title, authors, abstract, URL)
├── evidence/{Category}/{slug}.md     # Claims & benchmarks — fill this in
└── wiki/papers/{Category}/{slug}.md  # Synthesis & interpretation — fill this in
```

`sources/` files are written once and never touched again.
`evidence/` and `wiki/` files are only (re-)created when they still contain the unfilled template placeholder, so your edits are safe across re-runs.

## Using with Claude Code

Copy the skill file so `/arxiv-ingest` is available as a Claude Code command:

```bash
cp .claude/commands/arxiv-ingest.md ~/.claude/commands/
# or into your project's .claude/commands/
```

After running `arxiv-ingest run`, invoke `/arxiv-ingest` in Claude Code.
Claude reads each paper's abstract (and PDF when available) and fills in the `evidence/` and `wiki/` files automatically.

## GitHub Actions: daily auto-ingest

### Setup

1. Fork or use this repo as a template
2. Create a GitHub Fine-grained PAT with **Write** access to your wiki repo
3. Add to your repo's **Settings → Secrets and variables → Actions**:

   | Kind | Name | Value |
   |------|------|-------|
   | Secret | `GH_PAT` | your PAT |
   | Variable | `WIKI_REPO` | `your-name/research-wiki` |

4. Go to **Actions → Daily arXiv Ingest → Run workflow** to verify

### Schedule

- **Auto**: Mon–Fri UTC 01:00 (JST 10:00) — `cron: '0 1 * * 1-5'`
- **Manual**: `workflow_dispatch` with optional `days_back` override

## Requirements

- Python 3.11+
- [uv](https://docs.astral.sh/uv/) (recommended) or pip

## License

MIT
