Metadata-Version: 2.5
Name: ar2md
Version: 0.2.0
Summary: Save arXiv papers as LLM-ready Markdown, local images, and original PDFs
Project-URL: Homepage, https://github.com/cs-giung/ar2md
Project-URL: Repository, https://github.com/cs-giung/ar2md
Project-URL: Issues, https://github.com/cs-giung/ar2md/issues
Author: cs-giung
License-Expression: MIT
License-File: LICENSE
Keywords: arxiv,llm,markdown,pandoc
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Text Processing :: Markup
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# ar2md

Save a versioned arXiv paper as Markdown, local images, and the original PDF.
Markdown comes from native arXiv HTML, with ar5iv fallback after an HTTP 404.

## Requirements

- GNU Wget
- Pandoc 3.x for Markdown conversion; PDF downloading works without it

Set `WGET` or `PANDOC` to an executable path if needed.

## Usage

Run without installing:

```bash
uvx ar2md 1706.03762v7
```

Or install the command:

```bash
uv tool install ar2md
ar2md 1706.03762v7
```

IDs must include a version. Versioned arXiv `abs`, `html`, and `pdf` URLs are also
accepted; `1706.03762.v7` is normalized to `1706.03762v7`.

```bash
ar2md https://arxiv.org/abs/1706.03762v7 --output-dir refs
ar2md 1706.03762v7 --markdown-only  # skip the PDF download
```

To run an unreleased commit:

```bash
uvx --from git+https://github.com/cs-giung/ar2md ar2md 1706.03762v7
```

## Output

The default output directory is `papers/`:

```text
papers/
├── 1706.03762v7/       # local images, only when present
├── 1706.03762v7.md     # HTML-derived Markdown
├── 1706.03762v7.pdf    # original arXiv PDF
└── index.jsonl        # metadata, paths, warnings, and acquisition errors
```

Outputs are regular files, not symlinks. The HTML mirror and temporary resources
are removed afterward. Image paths are grouped under the versioned ID, including
ar5iv paths that originally omit the version. PDFs always come from
`https://arxiv.org/pdf/<versioned-id>` and must have a PDF header.

`index.jsonl` has one record per versioned ID, sorted by ID. Paths are relative to
the output directory; `source` identifies the HTML endpoint, and `html` is `null`
because HTML is not retained:

```json
{"id": "1706.03762v7", "title": "Attention Is All You Need", "source": "https://arxiv.org/html/1706.03762v7", "html": null, "markdown": "1706.03762v7.md", "pdf": "1706.03762v7.pdf", "warnings": [], "errors": {}}
```

When migrating from pre-0.2 mirror output, use a new output directory. Existing
symlinked output files and asset directories are refused; old mirrors are not deleted.

## Independent acquisition and refreshes

The PDF is attempted first, then Markdown. Failure of one does not suppress the
other attempt, including when Pandoc is missing:

| PDF | Markdown | Saved on a first attempt |
| --- | --- | --- |
| Success | Success | PDF, Markdown, images, and index |
| Success | Failure | PDF and index with a `markdown` error |
| Failure | Success | Markdown, images, and index with a `pdf` error |
| Failure | Failure | Index with both errors |

Unavailable formats have `null` paths on a first attempt; title and source may also
be `null`. On refresh, failed or skipped acquisitions retain their existing files
and indexed paths. A failed Markdown refresh also retains its previous title,
source, and warnings. `errors` records the latest failures by format.
`--markdown-only` neither downloads nor deletes a PDF.

Successful results and the index are staged before publication. If publication
fails, ar2md attempts to restore every replaced file. Failed or interrupted rollback
retains recoverable backups under `.ar2md-backup-*` in the output directory;
ordinary rollback errors report the location on stderr. Backups preserve relative
filenames. Stop other writes and keep these backups until recovery is complete.

An invalid existing index or unsafe destination can prevent all publication. This
is not crash recovery or support for concurrent writers. Unreferenced old assets
are not deleted.

Saved paths go to stdout; errors and quality warnings go to stderr. Exit code `0`
means every requested acquisition succeeded; `1` means an acquisition or publication
failed, even if a partial result was saved. Quality warnings alone do not fail the
command.

## Conversion and limits

The Lua filter selects the paper body, removes arXiv UI, preserves math, figures,
captions, and internal anchors, and reconstructs supported algorithms, footnotes,
and equation layouts. Image objects are prepared for Pandoc without rewriting
unrelated HTML. SVG assets remain external files.

Image discovery and path rewriting use Pandoc's syntax tree, including
reference-style images. Code examples and escaped image syntax remain literal;
HTML anchors remain HTML. No additional Markdown parser dependency is needed.

Checks flag unresolved citations and internal links, nested math delimiters,
missing local images, and selected source graphics omitted from Markdown. Counts
are printed and saved in `warnings`. These checks do **not** prove semantic
completeness or equivalence to the PDF. There is no PDF text extraction, OCR, TeX
fallback, or content fabrication. Network errors other than the native-HTML 404 do
not trigger ar5iv fallback; upstream HTML can change even for a fixed paper version.

## Development

```bash
uv sync --locked
uv run python -m unittest discover -s tests -v
uvx --from . ar2md 1706.03762v7
```

Pandoc-dependent tests are skipped when it is unavailable. PDF acquisition, CLI
error handling, and publication tests still run. CI installs Pandoc explicitly.
Saved artifacts and comparison methodology are documented in the
[comparison notes](https://github.com/cs-giung/ar2md/blob/main/tests/README.md).

## Publishing

Validate and rebuild from the current checkout, clearing stale distributions:

```bash
uv lock --check
uv run python -m unittest discover -s tests -v
uv build --clear
uvx twine check --strict dist/*
```

Publishing uses PyPI Trusted Publishing. Configure the `ar2md` project for GitHub
repository `cs-giung/ar2md`, workflow `publish.yml`, and environment `pypi`. Create a
GitHub release with a tag matching `v<project.version>` (`v0.2.0` for this version).
The workflow verifies the tag, runs tests, builds and validates distributions, then
publishes without a long-lived API token.

## License

MIT
