Metadata-Version: 2.4
Name: apibackuper
Version: 2.0.0
Summary: a command-line tool and python library for API backuping
Author-email: Ivan Begtin <ivan@begtin.tech>
License-Expression: MIT
Project-URL: Homepage, https://github.com/datacoon/apibackuper/
Project-URL: Documentation, https://ruarxive.org/apibackuper/
Project-URL: Repository, https://github.com/datacoon/apibackuper/
Project-URL: Issues, https://github.com/datacoon/apibackuper/issues
Project-URL: Download, https://github.com/datacoon/apibackuper/
Keywords: api,json,jsonl,csv,bson,cli,dataset,backup,archive
Classifier: Development Status :: 5 - Production/Stable
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Topic :: Software Development
Classifier: Topic :: System :: Networking
Classifier: Topic :: Terminals
Classifier: Topic :: Text Processing
Classifier: Topic :: Utilities
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: typer>=0.9.0
Requires-Dist: lxml>=4.9.3
Requires-Dist: urllib3>=2.0.6
Requires-Dist: requests>=2.31.0
Requires-Dist: xmltodict>=0.13.0
Requires-Dist: PyYAML>=6.0
Requires-Dist: jsonschema>=4.0.0
Requires-Dist: tqdm>=4.66.0
Requires-Dist: zstandard>=0.22.0
Provides-Extra: recommended
Requires-Dist: aria2p>=0.11.3; extra == "recommended"
Requires-Dist: pandas>=2.0.0; extra == "recommended"
Requires-Dist: pyarrow>=12.0.0; extra == "recommended"
Provides-Extra: dev
Requires-Dist: pre-commit>=3.6.0; extra == "dev"
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-mock>=3.12.0; extra == "dev"
Requires-Dist: pytest-timeout>=2.3.0; extra == "dev"
Requires-Dist: black==25.1.0; extra == "dev"
Requires-Dist: isort>=5.13.0; extra == "dev"
Requires-Dist: flake8>=7.0.0; extra == "dev"
Requires-Dist: flake8-docstrings>=1.7.0; extra == "dev"
Requires-Dist: flake8-bugbear>=23.12.0; extra == "dev"
Requires-Dist: flake8-comprehensions>=3.14.0; extra == "dev"
Requires-Dist: mypy>=1.8.0; extra == "dev"
Requires-Dist: types-requests>=2.31.0; extra == "dev"
Requires-Dist: types-PyYAML>=6.0.0; extra == "dev"
Requires-Dist: bandit>=1.7.6; extra == "dev"
Dynamic: license-file

# apibackuper

> A command-line tool to archive and backup REST APIs

**Version:** 2.0.0

This README describes version 2.0.0. See the
[changelog](CHANGELOG.md#200---2026-10-17) for release details and migration notes.

**apibackuper** downloads paginated REST APIs into local storage and exports
the archive as JSON Lines, gzip, zstd, or Parquet. It is designed for
repeatable, long-running backups rather than one-off HTTP requests.

## Features

- GET and POST iterative APIs (page, skip/offset, or range pagination)
- Full, incremental, and update work modes, with resume from checkpoints
- ZIP, filesystem, or SQLite storage
- Export to JSON Lines, gzip, Zstandard, or Parquet
- YAML configuration (INI still loaded, but deprecated)
- Authentication: Basic, Bearer, API Key, OAuth2
- Rate limiting, retries, concurrency, and SSL controls
- Follow-up requests for related objects and associated file downloads
- Optional Python hooks for custom pipeline steps

## Documentation

The full documentation site (Docusaurus) lives in [`docs/`](docs/) and is
published at **[ruarxive.org/apibackuper](https://ruarxive.org/apibackuper/)**.

| Section | What it covers |
|---------|----------------|
| [Getting started](https://ruarxive.org/apibackuper/getting-started/installation) | Install, quick start, positioning |
| [Cookbook](https://ruarxive.org/apibackuper/getting-started/cookbook) | Task index by goal |
| [CLI reference](https://ruarxive.org/apibackuper/commands/) | Every command |
| [Configuration](https://ruarxive.org/apibackuper/configuration/) | YAML sections and field reference |
| [Examples](https://ruarxive.org/apibackuper/examples/) | Working projects and templates |
| [Troubleshooting](https://ruarxive.org/apibackuper/getting-started/troubleshooting) | SSL, pagination, rate limits |

Source pages: [`docs/docs/`](docs/docs/). Changelog: [`CHANGELOG.md`](CHANGELOG.md).

## Installation

### Using pip

```bash
pip install --upgrade pip setuptools
pip install apibackuper
```

### Using pipx or uv

```bash
pipx install apibackuper
# or
uv tool install apibackuper
```

Parquet export and aria2 downloads:

```bash
pip install "apibackuper[recommended]"
```

### Requirements

- Python 3.10 or greater

### Install from source

```bash
python -m pip install --upgrade pip setuptools wheel
python -m pip install .
```

## Quick start

```bash
apibackuper create etrust
cd etrust
# save apibackuper.yaml with your API settings (and params.json for POST APIs)
apibackuper validate-config
apibackuper run full --dry-run
apibackuper estimate full
apibackuper run full
apibackuper export etrust.jsonl
```

A worked POST API example is in the
[quick start](https://ruarxive.org/apibackuper/getting-started/quick-start).
Starter YAML files are in [`examples/templates/`](examples/templates/).
`create` currently writes a minimal `apibackuper.cfg`; save your complete
configuration as `apibackuper.yaml`, which takes precedence over that INI file.
The API-related `create` flags are accepted for compatibility but do not populate
the configuration. Follow the [create reference](docs/docs/commands/create.md).

## Commands

```bash
apibackuper create <name>
apibackuper detect [--url URL] [--write-config]
apibackuper estimate full
apibackuper run full [--resume]
apibackuper run continue
apibackuper update [--resume]
apibackuper follow full|continue
apibackuper getfiles
apibackuper export output.jsonl
apibackuper info [--json]
apibackuper validate-config
```

`run full --dry-run` (alias `--profile`), `update --dry-run`, and
`follow full --dry-run` print a JSON plan without making HTTP requests.
`estimate full` contacts the API to measure a sample response.

`getfiles` currently reads ZIP backups. Its downloaded files can be stored in
ZIP or filesystem storage; see [file configuration](docs/docs/configuration/files.md).

```bash
apibackuper --help
apibackuper run --help
```

## Backup safety and recovery

A fresh `apibackuper run full` writes to a staging path ending in `.partial`
(for example, `storage.zip.partial`). It replaces the previous complete archive
only after every required page has been saved successfully. A failed full backup
keeps the previous archive and retains its staging data and checkpoint for recovery:

```bash
apibackuper run full --resume
```

Resume reuses saved pages and starts at the first unresolved page. Checkpoints
advance only through pages that have been saved durably. Failed or incomplete
backup runs exit with a nonzero status and preserve the last successful backup state,
so an update cannot advance its timestamp or change key past missing data.

Each `apibackuper update` retains its pages under a unique
`runs/<id>/page_N.json` generation. Use `apibackuper update --resume` to continue an
interrupted update within the same generation. Export reads the original pages
and every retained generation: it represents the retained archive history, and
does not deduplicate objects into a current snapshot. An object returned in several
updates can therefore appear several times in the export.

Existing archives with flat `page_N.json` members remain readable. Legacy
checkpoints without `next_page` are also accepted; resume safely replays their
last recorded page.

For ZIP storage, saved pages may also live in a `.journal` sidecar directory
until they are compacted into the archive. Keep `storage.zip.partial.journal`
beside `storage.zip.partial`, or `storage.zip.journal` beside `storage.zip`,
along with the checkpoint and state files, until recovery is complete. Resume
can finish an interrupted publication from saved data without downloading again.

`run continue` (alias `run incremental`) fills pages from the first missing
page in the published archive. To recover a staged full backup, use
`run full --resume`. See the [recovery guide](docs/docs/use-cases/incremental-and-update.md).

## Contributing

See the [development docs](https://ruarxive.org/apibackuper/development/contributing).

## Migration notes

### Python support

The minimum supported Python version is now **3.10**, declared by
`requires-python` in the package metadata. This is a breaking compatibility
change for Python 3.8 and 3.9 users; upgrade Python before installing this version.
CI covers Python 3.10–3.13 on Ubuntu and macOS.

### Storage backends (since v1.0.15)

The legacy `FileStorage`, `ZipFileStorage`, and `FilesystemStorage` API
in `apibackuper.storage` is **deprecated**. New code should use the unified
`StorageBackend` implementations from `apibackuper.storage.backends`:

| Legacy class              | Replacement                       |
| ------------------------- | --------------------------------- |
| `FileStorage`             | `StorageBackend` (base class)      |
| `ZipFileStorage`          | `ZipStorageBackend`               |
| `FilesystemStorage`       | `FilesystemStorageBackend`        |

The legacy concrete classes still work but emit `DeprecationWarning` on
instantiation. Importing alone is silent. Mapping the legacy API to the new
backend API:

```python
# Before (deprecated)
from apibackuper.storage import ZipFileStorage

storage = ZipFileStorage("pages.zip", mode="w")
storage.exists("page_1.json")           # -> bool
storage.store("page_1.json", b'{"id": 1}')
storage.close()

# After (preferred)
from apibackuper.storage import ZipStorageBackend

storage = ZipStorageBackend("pages.zip", mode="w")
"page_1.json" in storage.list_objects("page")   # -> bool
storage.save_page("page_1.json", b'{"id": 1}')
storage.close()
```

### YAML configuration

The INI config format is deprecated; YAML is the canonical format. A
YAML config supports `${VAR}` and `${VAR:-default}` placeholders for
sensitive credentials (see [SECURITY.md](SECURITY.md)).

## License

MIT License — see [LICENSE](LICENSE).

## Links

- [Documentation](https://ruarxive.org/apibackuper/)
- [GitHub](https://github.com/datacoon/apibackuper)
- [PyPI](https://pypi.org/project/apibackuper/)
- [Changelog](CHANGELOG.md)
- [Issue tracker](https://github.com/datacoon/apibackuper/issues)
