Metadata-Version: 2.4
Name: django-cloud-backup
Version: 0.1.0
Summary: Backs up django postgres databases, local folders and S3 folders to Google Drive, S3-compatible storage (AWS, Backblaze B2, Cloudflare R2) or Azure Blob Storage.
Author: Ian Jones
License-Expression: MIT
Project-URL: Homepage, https://github.com/jonesim/django-cloud-backup
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: encrypted-credentials
Requires-Dist: google-client-helper
Requires-Dist: requests
Provides-Extra: s3
Requires-Dist: boto3; extra == "s3"
Provides-Extra: azure
Requires-Dist: azure-storage-blob; extra == "azure"
Provides-Extra: encryption
Requires-Dist: cryptography; extra == "encryption"
Dynamic: license-file

[![PyPI version](https://badge.fury.io/py/django-cloud-backup.svg)](https://badge.fury.io/py/django-cloud-backup)


**django-cloud-backup** 

Backs up django postgres databases, local folders and S3 folders to Google Drive, S3-compatible storage (AWS, Backblaze B2, Cloudflare R2) or Azure Blob Storage.

**Migrating from django-gdrive-backup**

This package was previously published as `django-gdrive-backup`. The rename is a breaking
release; upgrading requires the following changes in your project:

- Install `django-cloud-backup` (and its extras, e.g. `django-cloud-backup[s3]`) instead of
  `django-gdrive-backup`
- `INSTALLED_APPS`: `'gdrive_backup'` → `'cloud_backup'`
- urls.py: `include('gdrive_backup.urls')` → `include('cloud_backup.urls')`, and any
  reverses/`{% url %}` tags use the `cloud_backup:` namespace instead of `gdrive_backup:`
- Celery beat schedules: task names are now `cloud_backup.tasks.*`
  (e.g. `cloud_backup.tasks.backup`)
- Settings renamed: `BACKUP_GDRIVE_DIR` → `BACKUP_ROOT`, `BACKUP_GDRIVE_DB` → `BACKUP_DB_DIR`
  (they apply to every destination backend, not just Google Drive). `BACKUP_TEAM_DRIVE` is
  unchanged.
- Removed legacy method aliases `backup_db_gdrive`, `restore_gdrive_db` and
  `restore_gdrive_folder` - use `backup_db_to_storage`, `restore_db_from_storage` and
  `restore_folder`
- Client-side encryption: the file format constants changed with the rename, so files
  encrypted by pre-release versions of the encryption feature cannot be read. No published
  release included encryption, so this affects no production backups.

Backups already in your storage destination are unaffected - folder layout and metadata
are unchanged, and existing backups restore as before.

**encrypted-credentials**

This package uses encrypted-credentials and the instructions there could be useful. Adding the following lines to **settings.py** will initialise the package

    from encrypted_credentials.django_credentials import add_encrypted_settings
    
    add_encrypted_settings(globals())


**Create service account**

Requires a Google service account with the Google Drive API enabled

https://console.cloud.google.com/apis/credentials/serviceaccountkey

**Add to cloud_backup installed apps**

settings.py

    INSTALLED_APPS = [ ..
            'cloud_backup',
        ]

**Store service account key**

By default *encrypted-credentials* is used to store the key. Create a directory off the django projects BASE_DIR called credentials and save the json key. 

settings.py

    CREDENTIAL_FOLDER = os.path.join(BASE_DIR, 'credentials')
    CREDENTIAL_FILES = {
        'drive': 'service-account.json',
    }
  

**Create Google Drive folder and share with service account**

With a Google Drive account create a folder and share with the email address of the service account.


**Ensure psql is available to python subprocess**

For docker containers you may need to something similar to the following line in the Dockerfile dependent on the version of Postgres.

    RUN apt-get -y install postgresql-client-11

**Configure database backup**

settings.py

    BACKUP_ROOT = 'django_backup'

**Choosing a backup destination**

Google Drive is the default destination and needs no extra settings beyond those above.
Backups can instead be stored on any S3-compatible service or Azure Blob Storage by adding
a `BACKUP_STORAGE` dict to settings.py. The optional `root` key replaces `BACKUP_ROOT`
as the top-level folder/prefix.

AWS S3:

    BACKUP_STORAGE = {
        'backend': 's3',
        'bucket': 'my-backups',
        'access_key_id': '...',
        'secret_key': '...',
        'root': 'django_backup',
    }

Backblaze B2 (the S3 endpoint is discovered from the key automatically):

    BACKUP_STORAGE = {
        'backend': 's3',
        'b2': True,
        'bucket': 'my-backups',
        'access_key_id': '...',       # B2 keyID
        'secret_key': '...',          # B2 applicationKey
    }

Cloudflare R2:

    BACKUP_STORAGE = {
        'backend': 's3',
        'bucket': 'my-backups',
        'endpoint_url': 'https://<account-id>.r2.cloudflarestorage.com',
        'region': 'auto',
        'access_key_id': '...',
        'secret_key': '...',
    }

Azure Blob Storage:

    BACKUP_STORAGE = {
        'backend': 'azure',
        'container': 'backups',
        'connection_string': '...',   # or account_url + credential
    }

S3 backends require `pip install django-cloud-backup[s3]` and Azure
`pip install django-cloud-backup[azure]`.

Note that unlike Google Drive, S3 and Azure destinations have no trash - pruned
database backups are deleted permanently, so consider enabling bucket versioning
(S3/B2/R2 lifecycle rules) or soft delete (Azure) if you want a safety net. The
backup page queries the destination and shows whether versioning, soft delete and
Object Lock/immutability (WORM) are actually enabled, so a missing safety net is
visible at a glance.

**Ransomware protection**

If backups re-sync whenever a source file changes, an attacker encrypting your files
would overwrite the good backups on the next scheduled run. Protection is layered:

*Changed-file handling* - settings.py:

    BACKUP_CHANGED_FILES = 'overwrite'   # default: re-upload changed files
    BACKUP_CHANGED_FILES = 'protect'     # never touch the existing backup: skip the
                                         # file, log a warning and fail the backup run
                                         # (raises ChangedFilesError after everything
                                         # else has completed, so monitoring alerts)
    BACKUP_CHANGED_FILES = 'history'     # keep the previous version (S3 server-side
                                         # copy named <file>.<timestamp>, Azure
                                         # snapshot, Google Drive rename), then upload
                                         # the new version; warns but succeeds

Use `'protect'` when your file store is immutable (e.g. UUID-named uploads that are
never edited) - any change is corruption or an attacker. Use `'history'` when changes
can be legitimate but you still want every previous version recoverable.

*Object Lock retention (S3 and B2)* - create the bucket with Object Lock enabled and
add a `lock` section to `BACKUP_STORAGE`:

    BACKUP_STORAGE = {
        'backend': 's3', ...,
        'lock': {
            'mode': 'COMPLIANCE',   # not even the bucket owner can shorten or delete
            'db_days': 35,          # each database dump is locked for 35 days at upload
            'file_days': 7,         # each file backup is locked for 7 days at upload
            'min_days': 7,          # extend_retention keeps every file locked >= 7 days ahead
        },
    }

Locking at upload costs nothing (extra headers on the existing request). To keep
long-lived file backups permanently locked, schedule the top-up task daily - it
extends any object whose remaining lock is below `min_days`:

    CELERY_BEAT_SCHEDULE = {
        'extend_retention': {
            'task': 'cloud_backup.tasks.extend_retention',
            'schedule': crontab(hour=3, minute=0),
        },
    }

or run `python manage.py backup_website --extend_retention`. The top-up costs 1-2 API
calls per file (roughly a minute per 10,000 files), which is why it is a scheduled
task rather than part of every backup. Backups only become deletable `min_days` after
the top-up task stops running.

Pruning still works on a locked bucket: Object Lock buckets are versioned, so deleting
an old dump just writes a delete marker (allowed even while versions are locked) and
the locked versions physically remain. Add a bucket lifecycle rule such as "expire
noncurrent versions after 40 days" to clean them up once their lock has passed. The
same versioning means even `'overwrite'` mode cannot physically destroy data on a
locked bucket - the prior locked version survives underneath.

*Credential and bucket hardening* (outside this package):

- **AWS S3**: give the backup IAM user no `s3:DeleteObject`/`s3:PutBucketLifecycle`;
  prune via lifecycle rules instead of `BACKUP_DB_RETENTION`
- **Backblaze B2**: use an application key without the `deleteFiles` capability
- **Azure**: enable blob soft delete or a container immutability policy
- **Google Drive**: deletes go to trash and are recoverable, but the service account
  can empty the trash - treat the credential file accordingly

With delete-less credentials, pruning logs a warning instead of failing the backup;
leave `BACKUP_DB_RETENTION` unset and let bucket lifecycle rules do the pruning.

**Client-side encryption**

By default backups are stored as the provider receives them - anyone with access to
the Drive folder or bucket can read a full database dump. Setting `BACKUP_ENCRYPTION`
encrypts every backup (database dumps, local folder files and S3-source files) on the
client before upload, using chunked AES-256-GCM, so the provider only ever holds
ciphertext:

    BACKUP_ENCRYPTION = True     # reuse the encrypted-credentials SETTINGS_KEY
    BACKUP_ENCRYPTION = '...'    # or a dedicated urlsafe-base64 32-byte key

`True` derives backup keys from the same `SETTINGS_KEY` that already protects the
encrypted credentials - nothing new to manage, and the key never sits in the
repository. The trade-off is coupling: today `SETTINGS_KEY` can be rotated cheaply by
re-encrypting the `.enc` settings files, but once backups are encrypted with it, old
backups need the old key forever. A dedicated key avoids that; generate one with:

    python -c "from encrypted_credentials.encrypted_file import random_key; print(random_key())"

and keep it in the encrypted private settings, not plain settings.py.

Notes:

- Restores are transparent - encrypted and older unencrypted backups are detected by
  content and both restore normally, including `restore_db --local_file`. A wrong or
  missing key fails cleanly before anything reaches `pg_restore`.
- **Losing the key means losing every backup encrypted with it.** Keep a copy of the
  key somewhere that does not depend on the server or the backups themselves.
- File deduplication keeps working: the plaintext md5 is recorded in each file's
  metadata at upload and compared on later runs. If you turn encryption off again,
  already-encrypted file backups re-upload once (their metadata is no longer fetched).
- Database backups briefly need twice the dump size in `BACKUP_LOCAL_DB_DIR` while the
  ciphertext copy is written; folder and S3-source backups encrypt in-stream with no
  extra disk.
- Requires the `cryptography` package (`pip install django-cloud-backup[encryption]`) -
  already present in practice, as encrypted-credentials depends on it.
- `BackupAzureToS3` is a separate rclone-compatible mirror and is not encrypted.
- With multiple backup configurations (below), encryption is set per config rather
  than globally.

**Multiple backup configurations**

`BACKUP_CONFIGS` lets one project back up to several destinations with different
behaviour per destination - the classic case being a hardened offsite backup plus an
unencrypted database copy that a staging server restores from:

    BACKUP_CONFIGS = {
        'default': {
            'storage': {'backend': 's3', 'bucket': 'offsite-backups', ..., 'lock': {...}},
            'encryption': True,
            'changed_files': 'protect',
        },
        'staging': {
            'storage': {'backend': 's3', 'bucket': 'staging-transfer', ...},
            'encryption': False,
            'dirs': [],                              # database only, no folder backups
            'retention': [{'days': 1, 'number': 2}], # keep just the latest couple of dumps
            'changed_files': 'overwrite',
        },
    }

Config keys: `storage` (a `BACKUP_STORAGE`-style dict), `encryption`, `db` (include
the database, default True), `db_dir`, `dirs` (as `BACKUP_DIRS`), `s3_dirs` (as
`S3_BACKUP_DIRS`), `retention`, `changed_files`. **A key absent from a config
inherits the corresponding legacy global setting** (`BACKUP_STORAGE`,
`BACKUP_ENCRYPTION`, `BACKUP_DIRS`, ...), so shared values can stay in the globals -
but note that means a config without `'dirs': []` backs up the global `BACKUP_DIRS`.
Without `BACKUP_CONFIGS` the globals simply are the default config, so existing
installations are unaffected.

Running a config:

    python manage.py backup_website --config staging
    python manage.py restore_db --config staging       # e.g. on the staging server

    CELERY_BEAT_SCHEDULE = {
        'backup': {'task': 'cloud_backup.tasks.backup',
                   'schedule': crontab(hour='8-19', minute=10)},
        'backup_staging': {'task': 'cloud_backup.tasks.backup',
                           'schedule': crontab(hour=6, minute=0),
                           'kwargs': {'config': 'staging'}},
    }

The web UI and un-parameterised tasks use the config named `'default'` (or the only
entry, if there is exactly one). For the staging pattern, the staging server's own
settings point a config at the same transfer bucket and `restore_db` pulls from it -
production never shares its offsite credentials or encryption key with staging. If
the transfer bucket holds an unencrypted production dump, treat the bucket itself as
production-sensitive, or give that config a dedicated key the staging server also
has.

Backups made by one config restore with that config's key: restoring an encrypted
backup through a config with a different key (or none) fails cleanly.


**Management commands**

    python manage.py backup_website
    python manage.py restore_db

**Management page**

urls.py

    urlpatterns = [
                    path('backup/', include('cloud_backup.urls')),
                    ....


All URL names live under the `cloud_backup` namespace (e.g.
`reverse('cloud_backup:backup-info')`). Previously the basic management page
used un-namespaced names such as `backup-info`; add the `cloud_backup:` prefix
if you reverse them yourself.

An enhanced version of the management page will be shown if the following django apps are installed

    'django_modals', 'django_datatables', 'django_menus', 'ajax_helpers'

from the following PyPi packages

    django-nested-modals, django-filtered-datatables, django-tab-menus, django-ajax-helpers

**Branding the management page**

The enhanced page views build the whole UI (menus, storage info and tables) into a
single HTML string, `{{ backup_content }}`, so it can be dropped into your own
template. Subclass the base views and set `template_name`:

    from cloud_backup.enhanced_views import BackupBaseView, SchemaTableBaseView

    class MyBackupView(BackupBaseView):
        template_name = 'myapp/backup.html'

    class MySchemaTableView(SchemaTableBaseView):
        template_name = 'myapp/backup.html'

The template must include the ajax_helpers/datatables/modals libraries and the
page script, then place the content wherever it fits your layout:

    {% load ajax_helpers %}
    {% lib_include 'ajax_helpers' 'Bootstrap' 'FontAwesome' module='ajax_helpers.includes' %}
    {% lib_include 'datatable' module='django_datatables.includes' %}
    {% lib_include 'Modals' module='django_modals.includes' %}
    {{ ajax_helpers_script }}
    ...
    {{ backup_content }}

Register the subclasses with `backup_urlpatterns` so the menu links and modals
(which reverse the standard `cloud_backup:` URL names) point at your views:

    from cloud_backup.urls import backup_urlpatterns

    urlpatterns = [
        path('backup/', include((backup_urlpatterns(
            backup_view=MyBackupView, schema_table_view=MySchemaTableView), 'cloud_backup'))),
    ]

The unbranded standard page remains the default when using
`include('cloud_backup.urls')`.

**Restoring from the management page**

Restore and drop-schema actions on the enhanced management page require

    BACKUP_ALLOW_RESTORE = True

which defaults to the value of `DEBUG`. This is enforced server-side on every
restore endpoint (not just by hiding the buttons), so a production server with
`DEBUG = False` and no `BACKUP_ALLOW_RESTORE` setting cannot be restored from
the web UI even by a superuser. Set `BACKUP_ALLOW_RESTORE = True` on staging and
development machines where restoring is wanted. The `manage.py restore_db`
command is not affected by this setting, so disaster recovery on a live server
remains possible from the command line.


**Configure S3 folder backups**

settings.py

            AWS_ACCESS_KEY_ID = id
            AWS_SECRET_ACCESS_KEY = key
            AWS_PRIVATE_STORAGE_BUCKET_NAME = bucket
            
            S3_BACKUP_DIRS = [('S3-source-folder1', 'google-drive-folder1'),
                              ('S3-source-folder2', 'google-drive-folder2')
            ]
            
**Configure cleaning of old datatabase backups**

settings.py

    BACKUP_DB_RETENTION = [{'hours': 1, 'number': 4}, 
                           {'hours': 2, 'number': 10},
                           {'days': 1, 'number': 10},
                           {'months': 1, 'number': 36},
                           ]

             
**Schedule backup with celery beat**

    CELERY_BEAT_SCHEDULE = {
        'backup': {
            'task': 'cloud_backup.tasks.backup',
            'schedule': crontab(hour='8-19', minute=10, day_of_week='mon-fri')
        }
    }
            
