Metadata-Version: 2.4
Name: declarative-data-backup
Version: 0.3.0
Summary: Declarative Borg-based data backups
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: PyYAML<7,>=6

# Declarative Data Backup

Declarative Data Backup creates encrypted Borg archives for configured source
directories. Each source gets its own archive series in one Borg repository,
so Borg can deduplicate chunks across all sources. The application validates
the complete configuration before invoking Borg and prunes a source's archives
only after every create operation in the run has succeeded.

## Requirements

- Python 3.11 or newer.
- Borg 1.4.x installed separately and available as `borg` on `PATH`.
- A Borg repository that has already been initialized in keyfile encryption
  mode, its matching keyfile, and a protected passphrase source.
- For remote repositories, OpenSSH and a trusted `known_hosts` file.

The application never initializes a repository. Borg 2 is not supported.

## Installation

Install the package from PyPI into a `uv`-managed environment:

```shell
uv venv
uv pip install declarative-data-backup
```

For a checkout, install the development dependencies with:

```shell
make install
```

To also install the repository's pre-commit hooks, run `make setup`.
The PyPI distribution is named `declarative-data-backup`; its importable
package is `declarative_data_backup`.

## GitLab release automation

The pipeline in [.gitlab-ci.yml](.gitlab-ci.yml) publishes releases when a Git
tag in `major.minor.patch` format is pushed, for example `1.2.3`. Use the same
version for the GitLab Release name and tag. Tags with a `v` prefix, prerelease
suffix, or leading zeros are not published. Branch pushes do not run this
release pipeline.

For each version tag, the pipeline:

1. Sets the package version in the CI checkout's `pyproject.toml` and uses
   `uv build` to create a wheel and source distribution.
2. Uploads `declarative-data-backup` to PyPI.
3. Builds the existing Python 3.13 Dockerfile with the same package version,
   verifies the installed version, and pushes
   `${CI_REGISTRY_IMAGE}/declarative-data-backup:1.2.3-python3.13` to the
   project's GitLab Container Registry.

The source checkout's version is not committed back to the repository.
The two publishing jobs run independently after the package build succeeds;
publishing is not atomic across PyPI and the container registry.

Before the first release:

- Enable the project's GitLab Container Registry.
- Add `PYPI_API_TOKEN` in GitLab **Settings > CI/CD > Variables** as a masked,
  protected variable. Use a PyPI API token authorized to publish
  `declarative-data-backup`; for a new PyPI project, its first upload may
  require an account-wide token, which should then be replaced with a
  project-scoped token.
- Protect release tags in GitLab (for example, `*.*.*`) and restrict who can
  create them. Protected variables are only available on protected refs.
- Use a trusted Docker-executor runner configured for Docker-in-Docker with
  TLS: `privileged = true` and `volumes = ["/certs/client", "/cache"]` in its
  `[runners.docker]` configuration. GitLab.com instance runners support this
  setup. Privileged runners should not execute untrusted code. If your runner
  requires a tag, add it to `publish_container`.

Registry authentication uses GitLab's built-in `CI_REGISTRY_USER` and
`CI_REGISTRY_PASSWORD`; no separate registry password is needed.

Create and push a release tag from the commit you want to publish:

```shell
git tag 1.2.3
git push origin 1.2.3
```

Creating a GitLab Release with a new `1.2.3` tag also triggers the tag pipeline.
Creating or editing a Release on an already existing tag does not trigger a
new pipeline. PyPI versions cannot be overwritten: do not move or reuse
release tags. If only the container job fails, retry that job rather than
rerunning a successful PyPI upload.

## Configure and run

Create a YAML file accessible to the backup process:

```yaml
media_backup:
  folders:
    - /server_root/media/photos
    - /server_root/media/music
  compression: auto,zstd,22
  retention_days: 30
  required_mounts:
    - /server_root/media
  allow_empty: true

postgres_backup:
  folders:
    - /server_root/path/to/postgres-backups
  required_mounts:
    - /server_root/path/to/postgres-backups

apprise_backup:
  files:
    - /server_root/data/apprise/apprise.yml

jellyfin_backup:
  folders:
    - /server_root/data/jellyfin
  exclude:
    - "**/cache/**"
    - "**/log/**"
    - "**/transcodes/**"
```

Set the required environment variables before starting the process:

| Variable | Purpose |
| --- | --- |
| `BACKUP_CONFIG_PATH` | Absolute path to the YAML configuration. |
| `SERVER_ROOT` | Absolute path to the root containing all configured source directories. |
| `BORG_REPO` | Existing Borg repository location, local or remote. |
| `BORG_KEY_FILE` | Path to the external keyfile for the encrypted repository. |
| `BORG_BASE_DIR` | Writable directory for Borg state and the backup coordination lock. |
| `BORG_RSH` | SSH command and options used by Borg for a remote repository. |
| `BORG_PASSCOMMAND` | Command Borg runs to obtain the repository passphrase. |

Optional variables:

| Variable | Purpose |
| --- | --- |
| `BORG_REMOTE_PATH` | Remote Borg executable path, passed as Borg's `--remote-path`. Leave unset to use Borg's default. |
| `BORG_EXIT_CODES` | Borg exit-code mode: `legacy` or `modern`. |
| `BACKUP_PROGRESS` | Set to `true` to enable Borg's live progress output during archive creation. Defaults to `false` to limit log volume. |
| `BACKUP_VERBOSE` | Set to `true` to show Borg's per-file create and per-archive prune output. Defaults to `false` for concise logs. |

Provide credentials through a secret manager or protected files; do not put
passphrases, private keys, or credentials in the YAML file, source control, or
shell history. For example, in a controlled environment where these variables
have already been securely populated:

```shell
python -m declarative_data_backup
```

The command exits with status `0` on success, nonzero on configuration or Borg
failure, and `130` when interrupted. Borg's exit code `1` indicates warnings:
the run fails and pruning is skipped. Logs include group and source start/end
messages, while Borg's per-file and per-archive output is disabled by default.
Set `BACKUP_VERBOSE=true` for detailed Borg output or `BACKUP_PROGRESS=true`
for live progress during archive creation; either can increase log volume.

## Configuration reference

The top level must be a nonempty YAML mapping of backup group names to group
mappings. Group names may contain ASCII letters, digits, `_`, and `-`, and
must be at most 64 characters. Duplicate YAML keys, unknown group fields,
invalid types, unsupported pattern selectors, and empty configuration are
rejected before any create or prune operation.

Each group supports:

- `folders` (optional): list of absolute directory paths.
- `files` (optional): list of absolute regular-file paths. Folders and files
  share the same containment, readability, duplicate, and overlap checks;
  configure at least one source per group.
- `exclude` (optional): list of Borg 1.4 glob patterns, scoped to this group
  and passed directly to Borg for folder sources. To avoid approximating
  Borg's pattern matching, a group with explicit `files` cannot also configure
  `exclude`; this is rejected before any Borg operation.
  Patterns are otherwise passed unchanged to Borg, which evaluates their
  matching semantics. If Borg rejects a pattern during create, the run stops
  before pruning any source.
- `compression` (optional): defaults to `auto,zstd,22`. Supported Borg 1.4
  codecs include `none`, `lz4`, `zstd`, `zlib`, and `lzma`; `auto` may precede
  a codec. Codec levels are checked against supported ranges: zstd `1`-`22`,
  zlib `0`-`9`, and lzma `0`-`9`. Borg's `obfuscate` compression specifier is
  also accepted for specifier values `1`-`6`, `110`-`123`, or `250`; it is
  useful only with repository encryption.
- `retention_days` (optional): positive integer, default `30`. This controls
  Borg's `--keep-daily` policy.
- `required_mounts` (optional): list of exact mount points that must appear
  in the process's `/proc/self/mountinfo`.
- `allow_empty` (optional): boolean, default `true`. If set to `false`, an
  empty source is rejected.

Configure `required_mounts` explicitly for every source that depends on a
separately mounted filesystem. The program does not infer mount requirements
from source paths. In a container or Kubernetes pod, mounting `/server_root`
does not prove that nested host filesystems are mounted. In particular,
`allow_empty: true` does not prove that a disk is present: an empty source may
be legitimate and can be archived.

Backing up a live SQLite database with `files` or `folders` does not produce a
consistent database snapshot. Use an application-supported export, backup API,
or coordinated filesystem snapshot for live database data. Exclusions do not
make a database backup consistent.

Paths are resolved before backup. A top-level source symlink is allowed only
when its canonical target remains inside `SERVER_ROOT`; symlinks within a
source are stored as links rather than followed.

## Archives, retention, and compaction

The application creates one archive per configured source in the single
`BORG_REPO`. Names have this form:

```text
bbkp__<source-series>__<UTC timestamp>__<run UUID>
```

The stable source series is derived from the canonical source path and includes
a SHA-256 suffix. Borg receives the canonical absolute path as the source, so
the original directory hierarchy is retained inside the archive.

The application creates all configured archives before pruning any source.
If a create fails or Borg reports warnings, the run stops and no pruning is
performed. Retries use a new run UUID and may create another archive for that
day; archives are never overwritten to handle a collision.

`--keep-daily N` retains the latest archive in up to `N` daily buckets. It does
not guarantee `N` successful calendar days of recovery if scheduled runs fail
or are missed. Removing a source from the YAML does not delete its old
archives. The source-series selector does not match archives made with the
previous non-namespaced naming scheme; retain those archives unless a
separately reviewed migration or cleanup is performed.

Pruning marks repository data for deletion; it does not reclaim the space.
Repository compaction is a separate maintenance operation and is not run by
this application. Any scheduled compact job or manual maintenance must first
acquire an exclusive `flock` on:

```text
<BORG_BASE_DIR>/backup.lock
```

Use the same persistent state volume and filesystem as the backup job, with
working advisory-lock semantics. The backup process waits up to 3600 seconds
for this lock. Kubernetes `concurrencyPolicy: Forbid` does not coordinate
separate CronJobs, and Borg's own repository lock does not replace this
application-level lock. Never unlink or replace the lock file while any
process could hold it.

## Deployment and operational notes

The Dockerfile builds an image with Borg 1.4.1 and the Python package. The
application itself does not set CPU or I/O priority; a scheduled deployment
may wrap the command with `nice -n 19` and `ionice -c 3`. For Kubernetes or
another container runtime:

- Run as a non-root user and confirm it can read the keyfile and credentials
  and write the Borg state volume.
- Mount source data read-only. Keep Borg cache, configuration, security state,
  keyfile, and the coordination lock on persistent writable storage.
- For nested source filesystems, configure and verify the corresponding
  `required_mounts`.
- Use a versioned image tag or digest and confirm private-registry pull
  credentials have only the required read access.
- Do not mount a service-account token unless the workload needs one.
- Ensure any compact job mounts the same state volume and honors the lock
  contract above.
- Monitor failed and warning backup jobs. Retention cannot make up for days
  without successful archives.
- A fixed schedule does not ensure upstream jobs, such as Unmanic processing,
  have completed; establish that dependency separately.

The default `auto,zstd,22` may be slow or exceed a 1 GiB memory limit on some
data sets. It has not been benchmarked under that limit and may be terminated
for memory use. Compression can be overridden per group without recreating
the repository.

Kubernetes Secret projection permissions, storage ownership, mount behavior,
node placement, registry access, and workload scheduling must be verified in
the target cluster. They are not verified by this Python package or its
container integration test.

## Testing

Run the unit tests, integration tests, or both:

```shell
make test-unit
make test-integration
make test
```

The integration target opts in to both local Borg tests (which use temporary
encrypted repositories) and the Docker SSH test (which uses disposable
containers and a temporary repository). It does not use production credentials
or BorgBase. Docker must be installed and its daemon available.

To run either integration suite individually:

```shell
RUN_BORG_LOCAL_INTEGRATION=1 uv run pytest -m integration \
  tests/integration/test_local_borg_integration.py
RUN_BORG_DOCKER_INTEGRATION=1 uv run pytest -m integration \
  tests/integration/test_docker_integration.py
```

Do not treat skipped integration tests as successful restore validation.

## Recovery

Recovery requires the repository and the matching external keyfile and
passphrase. Keep independent recovery copies of the keyfile and passphrase;
the repository must not be their only copy. Also retain the SSH private key
and verified `known_hosts` entry for a remote repository.

With the required Borg environment and credentials securely configured, list
archives and inspect the desired archive:

```shell
borg list "$BORG_REPO"
borg list "$BORG_REPO"::'bbkp__<source-series>__<timestamp>__<run-uuid>'
```

Extract into an isolated recovery directory and verify the restored data
before returning it to service:

```shell
mkdir -p /restore
cd /restore
borg extract "$BORG_REPO"::'bbkp__<source-series>__<timestamp>__<run-uuid>'
```

Test recovery periodically using a disposable or approved recovery
environment. This repository's tests do not verify access to a live remote
repository or actual Kubernetes Secret projection and storage behavior.
