Metadata-Version: 2.4
Name: paperpulse
Version: 0.2.0
Summary: Relevance-ranked arXiv digests: the few papers that actually matter to you, summarised.
Author: viki22uied
License: MIT
Project-URL: Homepage, https://github.com/viki22uied/PaperPulse
Project-URL: Repository, https://github.com/viki22uied/PaperPulse
Project-URL: Issues, https://github.com/viki22uied/PaperPulse/issues
Keywords: arxiv,research,digest,nlp,embeddings,recommender
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Linguistic
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy<3,>=1.23
Requires-Dist: scikit-learn<2,>=1.1
Requires-Dist: pyyaml<7,>=6.0
Provides-Extra: semantic
Requires-Dist: sentence-transformers<6,>=2.2; extra == "semantic"
Provides-Extra: llm
Requires-Dist: openai<2,>=1.0; extra == "llm"
Requires-Dist: anthropic<1,>=0.30; extra == "llm"
Provides-Extra: pdf
Requires-Dist: pypdf<6,>=4.0; extra == "pdf"
Provides-Extra: dev
Requires-Dist: pytest<10,>=7.0; extra == "dev"
Provides-Extra: all
Requires-Dist: sentence-transformers<6,>=2.2; extra == "all"
Requires-Dist: openai<2,>=1.0; extra == "all"
Requires-Dist: anthropic<1,>=0.30; extra == "all"
Requires-Dist: pypdf<6,>=4.0; extra == "all"
Dynamic: license-file

# PaperPulse

[![CI](https://github.com/viki22uied/PaperPulse/actions/workflows/ci.yml/badge.svg)](https://github.com/viki22uied/PaperPulse/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/paperpulse)](https://pypi.org/project/paperpulse/)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://www.python.org/)
[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)

**The five papers that actually matter to you today — ranked, trust-scored, and summarised in plain English.**

📄 **[See a real digest generated today →](examples/2026-07-14.md)** &nbsp;·&nbsp;
🗺️ **[Roadmap](ROADMAP.md)** &nbsp;·&nbsp;
🧪 **[Annotated sample](examples/sample-digest.md)**

arXiv drops hundreds of papers a day. Summaries are a solved problem; *triage*
isn't. PaperPulse is built around the two questions that actually waste your
time:

1. **Which of today's 200 papers are relevant to *me*?** — a learnable interest
   profile ranks every abstract by similarity to what you care about, and a
   thumbs-up/down feedback loop sharpens it over time.
2. **Which of those should I actually trust?** — a set of offline *signal-quality*
   heuristics flag over-claiming, missing error bars, weak baselines, likely data
   leakage, benchmark-chasing, and incremental "me-too" work, before you sink
   twenty minutes into the PDF.

It runs out of the box with **no API keys and no model downloads** (local hashing
embeddings + extractive summaries), and scales up cleanly to semantic embeddings
and LLM summaries when you want them.

```bash
pip install paperpulse    # or: pip install -e .  (from a clone)
paperpulse init           # write a starter paperpulse.yaml
# edit `interests` and `categories`
paperpulse run            # today's ranked, trust-scored digest -> digests/YYYY-MM-DD.md
```

The [sample digest](examples/sample-digest.md) is annotated to show the finance
paper getting a 🚩 *leakage* flag and the hype paper collecting four flags while a
careful LoRA study rises to the top.

---

## Why it's not just another arXiv scraper

| | Typical "arXiv + LLM" script | PaperPulse |
|---|---|---|
| Ranking | none / recency | learnable interest vector + **MMR diversity** |
| Personalisation | static keywords | **Rocchio feedback loop**, per-user profiles |
| Trust | blind trust in the summary | **10+ signal-quality checks** with a badge |
| Novelty | — | **crowding score** flags incremental work |
| Cost to start | API key required | **runs fully offline** |

## How the ranking works

1. **Embed** each abstract into a stable vector space. The default
   `HashingBackend` needs no model and no training, so a profile learned today is
   still comparable next week. Install `paperpulse[semantic]` to switch to
   sentence-transformers for sharper semantics — same interface, better vectors.
2. **Score** by cosine similarity to your interest vector.
3. **Select** the top *N* with **Maximal Marginal Relevance**, trading a little
   relevance for variety so you don't get five near-identical papers.
4. **Learn.** `paperpulse feedback --like 2407.00004 --dislike 2407.00002`
   nudges the profile toward what you liked and away from what you didn't
   (Rocchio), anchored so it never drifts far from the interests you wrote down.

## The trust layer

Working from a title and abstract we can't *prove* a paper is sound — but we can
cheaply flag the patterns that correlate with weaker work. Each signal is
deterministic, explains itself, and rolls up into a 🟢 clean / 🟡 mixed /
🔴 caution badge. Any hard red flag keeps a paper out of "clean".

Every flag is **legible**: it shows the *exact phrase that tripped it* and a
*confidence* so you can dismiss weak heuristics at a glance, not just a colour.
A **peer-review / venue** signal separates published or venue-accepted work from
preprint-only (and flags papers that have sat on arXiv for years, still v1, with
no venue). Each paper also carries a **worth-it** score — relevance × trust — to
answer "is this worth my next 30 minutes?"

Several are drawn straight from systematic-trading research discipline:

- **crowding / novelty** — is the method a cosmetic rehash of neighbours in the same batch?
- **subgroup robustness** — do strong aggregate numbers hide weak subgroups?
- **metric gaming** — did a metric move without a genuine underlying gain?
- **deployability** — oracle inputs, look-ahead features, or unrealistic compute?
- **leakage** — random splits on time-series/financial data (a classic lookahead trap)
- **backtest overfitting** — backtested returns with no out-of-sample, walk-forward, or transaction-cost check
- **baseline fairness**, **evidence / error bars**, **over-claiming**, **benchmark saturation**, **reproducibility (code/data links)**

Turn on `--online` (or `trust_online: true`) to add **dead-link detection**, a
**Retraction Watch** cross-check, and a **self-citation ratio** (via Semantic
Scholar — set `S2_API_KEY` for higher rate limits).

## Beyond the digest

- **Multiple sources** — arXiv, bioRxiv/medRxiv, and PubMed behind one interface (`paperpulse run --source pubmed`). Adding OpenReview/SSRN is one adapter.
- **Contradiction mapping** — surfaces pairs of closely-related papers that report opposing outcomes.
- **Cross-reference your own work** — `paperpulse similar my_model.py` finds papers whose methods are functionally closest to your code or notebook.
- **Market context for finance papers** — when a paper names a well-known asset (an index, crypto, commodity, or mega-cap), the dashboard tags it with the latest price from Yahoo Finance so you can sanity-check a claim at a glance. Stdlib-only, no API key.
- **Delivery anywhere** — Markdown file, email (SMTP), RSS feed, or Slack/Discord webhook.
- **Community trust store** — a self-hostable SQLite DB that pools trust reports across users and builds an over-claiming leaderboard.

## Three ways to use it

```bash
# 1. CLI
paperpulse run --source arxiv --categories cs.LG cs.CL

# 2. Python SDK
python -c "from paperpulse.pipeline import run_digest; from paperpulse.config import Config; \
           print(run_digest(Config(), dry_run=True).markdown)"

# 3. REST API + web dashboard (stdlib only, no extra deps)
paperpulse serve            # http://127.0.0.1:8000
#   Interactive topic-filter bar (Finance / Economics / Quant), loads instantly,
#   with live price chips on any paper that names a major asset.
#   GET /api/digest?cats=q-fin.TR,econ.GN   POST /api/feedback   GET /api/community/leaderboard
```

Or with Docker:

```bash
docker compose up          # dashboard on :8000, state persisted in a volume
```

## Configuration

`paperpulse init` writes a commented `paperpulse.yaml`
(see [`paperpulse.example.yaml`](paperpulse.example.yaml)). Everything is
config-driven: sources, categories, which trust signals to enable, ranking
diversity, and delivery.

Secrets are read from the environment, never the config file:
`OPENAI_API_KEY` / `ANTHROPIC_API_KEY` (LLM summaries), `PAPERPULSE_SMTP_*`
(email), `PAPERPULSE_SLACK_WEBHOOK` / `PAPERPULSE_DISCORD_WEBHOOK`, `NCBI_API_KEY`.

## Scheduling

Run it daily with cron, or use the included GitHub Action
([`.github/workflows/digest.yml`](.github/workflows/digest.yml)) to generate and
commit a digest every weekday morning.

## Development

```bash
pip install -e ".[dev]"
pytest
```

The full test suite is offline — no network, no keys — so it runs anywhere.

See [`ROADMAP.md`](ROADMAP.md) for what's shipped and what's next.

### Publishing

`pip install build twine && python -m build && twine check dist/*` builds and
validates the wheel. Cutting a **GitHub Release** then publishes it to PyPI via
[`release.yml`](.github/workflows/release.yml) (Trusted Publishing — no token in
the repo; add the publisher once on PyPI first). Or push manually with
`twine upload dist/*`.

## License

MIT — see [`LICENSE`](LICENSE).
