Metadata-Version: 2.4
Name: arxiv-sectionizer
Version: 0.1.0
Summary: Extract semantic sections (future work, limitations, conclusion, ...) from arXiv LaTeX sources.
Project-URL: Homepage, https://github.com/danielquillanr/arxiv-sectionizer
Project-URL: Issues, https://github.com/danielquillanr/arxiv-sectionizer/issues
Author-email: Daniel Quillan <danielquillanr@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: arxiv,latex,nlp,papers,parser,research,section
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Text Processing :: Linguistic
Classifier: Topic :: Text Processing :: Markup :: LaTeX
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# arxiv-sectionizer

Extract semantic sections — **Future Work**, **Limitations**, **Conclusion**, **Discussion**, etc. — from arXiv LaTeX source archives. Zero runtime dependencies.

The `arxiv` package hands you metadata and a PDF URL. This library hands you the *sections you actually want*, pulled from LaTeX source with `\input{}` / `\include{}` / `\subfile{}` includes resolved.

## Install

```bash
pip install arxiv-sectionizer
```

## Quickstart

```python
from arxiv_sectionizer import sectionize

# Accepts a tar.gz path, raw bytes, a .gz, or a .tex string.
for s in sectionize("2401.01234.tar.gz", kinds="future_work"):
    print(s.title)
    print(s.body[:300])
```

Want every section? Omit `kinds`:

```python
sections = sectionize("paper.tar.gz")
for s in sections:
    print(s.kind or "-", s.level, s.title)
```

## Lower-level API

```python
from arxiv_sectionizer import read_archive, flatten, extract_sections, find_sections

files = read_archive("paper.tar.gz")          # dict[str, str] of every .tex member
tex = flatten(files)                          # resolve \input{}/\include{}/\subfile{}
sections = extract_sections(tex)              # all sections, with .kind classification
future = find_sections(tex, "future_work")    # only matching categories
```

Each `Section` is:

```python
Section(
    title: str,
    body: str,
    level: int,          # 0=chapter, 1=section, 2=subsection, ...
    kind: str | None,    # "future_work" | "limitations" | "conclusion" | ...
    start: int,          # byte offsets into the flattened source
    end: int,
)
```

## Custom categories

The default category map covers the usual targets. Override it with your own keyword map:

```python
from arxiv_sectionizer import Categories, extract_sections

cats = Categories(mapping={
    "open_problems": ("open problem", "unsolved"),
    "ablation":      ("ablation",),
})
sections = extract_sections(tex, categories=cats, only_classified=True)
```

Default categories: `future_work`, `limitations`, `conclusion`, `discussion`, `related_work`, `introduction`, `abstract`, `methods`, `results`.

## CLI

```bash
arxiv-sectionize paper.tar.gz --kind future_work,limitations
arxiv-sectionize paper.tar.gz --titles-only
arxiv-sectionize paper.tar.gz --json > sections.json
```

## Why

Every paper-mining pipeline re-implements "give me this section from this paper" over LaTeX, and every implementation re-discovers the same edge cases:

- nested `\subsection` / `\paragraph`
- starred forms (`\section*{...}`)
- titles with inline LaTeX commands (`\section{Results on \textsc{LibriTTS}}`)
- split-file papers using `\input{conclusion}`
- comment-outed sections (`% \section{Old}`)
- PDF-only submissions with no .tex source

This library handles those once.

## License

MIT
