Metadata-Version: 2.4
Name: df-s3-filen-wrapper
Version: 0.1.1
Summary: A correctly configured boto3 client for the Filen CLI's S3 gateway, plus a live probe that checks its quirks.
Keywords: filen,s3,boto3,object-storage
Author: Justin Guese
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: System :: Archiving
Classifier: Typing :: Typed
Requires-Dist: boto3>=1.36
Requires-Python: >=3.10
Project-URL: Homepage, https://github.com/JustinGuese/df-s3-filen-wrapper
Project-URL: Issues, https://github.com/JustinGuese/df-s3-filen-wrapper/issues
Description-Content-Type: text/markdown

# df-s3-filen-wrapper

A correctly configured boto3 client for the [Filen](https://filen.io) CLI's S3 gateway
(`filen s3`), a few helpers for the gaps that configuration can't close, and a probe that
checks a live gateway still behaves the way this README says.

```bash
pip install df-s3-filen-wrapper        # or: uv add df-s3-filen-wrapper
```

> **Don't have Filen storage yet?** It's end-to-end encrypted, S3/FTP/WebDAV compatible
> (this package uses the S3 gateway), and considerably cheaper than S3/Backblaze/Drive for
> the same space. Sign up with
> [my referral link](https://filen.io/r/33aba21d9edfbb38c0b1e222fdd52521) and you get 10GB
> free.

## Why this exists

The obvious way to use the gateway is the usual code for any S3-compatible endpoint:

```python
boto3.client("s3", endpoint_url="https://filen-s3.example.com", ...)
```

**With a current boto3, every upload made that way is silently corrupted.** botocore
≥ 1.36 sends uploads with `Content-Encoding: aws-chunked` framing and a CRC32 trailer. The
gateway doesn't decode that framing and stores it as file content, so a 4096-byte file
lands as 4140 bytes. The PUT returns 200 and nothing raises; you find out when something
reads the file.

The gateway implements only part of S3, and several other gaps are just as easy to miss.
Each one below was measured against a live gateway, and the probe re-checks every one that
can be checked from outside.

## Usage

```python
from pathlib import Path

from df_s3_filen_wrapper import FilenBucket, make_client

s3 = make_client(
    "https://filen-s3.example.com",   # or FILEN_ENDPOINT
    "access-key",                     # or FILEN_ACCESS_KEY
    "secret-key",                     # or FILEN_SECRET_KEY
)                                     # region: FILEN_REGION, default "filen"

bucket = FilenBucket(s3, "my-bucket")  # or FilenBucket.from_env() with FILEN_BUCKET
bucket.ensure()

bucket.put("videos/2026/a.mp4", Path("a.mp4"), content_type="video/mp4")  # streamed
data = bucket.get("videos/2026/a.mp4")
bucket.download("videos/2026/a.mp4", "/tmp/a.mp4")   # streamed, atomic
bucket.exists("videos/2026/a.mp4")                    # True
bucket.stat("videos/2026/a.mp4")                      # Entry(key, size, last_modified)
bucket.list("videos/")                                # no directory markers
bucket.usage("videos/")                               # total bytes
bucket.delete("videos/2026/a.mp4")
bucket.delete_prefix("videos/2026/")                  # objects AND markers
```

`make_client` returns a **plain boto3 client**; nothing is wrapped. `FilenBucket` covers
the operations that need a workaround, and `bucket.client` is the same client for
everything else. To add your own botocore settings, pass `config=Config(...)`. It is
merged in, but the settings the gateway requires are always applied last.

## What the gateway does differently

| #   | Behaviour                                                                                                                                    | Consequence                                                       | Handled by                                                    |
| --- | -------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------- |
| 1   | Stores `aws-chunked` framing verbatim                                                                                                        | Uploads silently corrupted                                        | `make_client`: `request_checksum_calculation="when_required"` |
| 2   | No virtual-host addressing; won't sign without a region                                                                                      | Requests fail                                                     | `make_client`: path style, region `"filen"`                   |
| 3   | No multipart upload (`501`)                                                                                                                  | Only single PUTs; 64 MiB measured working                         | `put` refuses anything over `MAX_OBJECT_BYTES` before sending |
| 4   | Rejects a `Range` ending at or past the object size (`400`)                                                                                  | boto3's `download_file` fails on every object under 8 MiB         | `get` / `download` use `get_object`                           |
| 5   | `ListObjectsV2` without `Prefix` is a `400`; `MaxKeys` ignored                                                                               | Listing the bucket fails                                          | `list` always sends `Prefix`                                  |
| 6   | Zero-byte **directory markers** (`a/`, `a/b/`) for every path segment, left behind on delete; a marker shows up only when listing its parent | Wrong key counts and byte totals; "deleted" directories linger    | `list` drops them; `delete_prefix` removes them               |
| 7   | `HeadObject` answers **`401`** for about a second after a PUT, then `200` (a missing key gets a correct `404`)                               | A HEAD-based `exists()` reports a fresh object as an auth failure | `exists` / `stat` use a listing                               |
| 8   | `x-amz-meta-*` accepted on PUT, never returned; no `ETag` on GET                                                                             | Metadata is lost                                                  | Not fixable. Keep metadata in your own database               |
| 9   | Anonymous and presigned GETs both answer `400 Required headers are missing` (header SigV4 only)                                              | Browsers can't load objects; no `<img src>` / `<video src>`       | Not fixable. Serve through your own backend or a CDN/YouTube  |

**Behind Cloudflare** (e.g. a tunnel), Python's default `urllib` User-Agent gets
`403 error code: 1010` from Cloudflare before the request reaches the gateway. That can
look like the gateway refusing a request when it never saw it. boto3's own User-Agent gets
through.

## The probe

The gateway is whatever Filen CLI version your deployment is running. If that image isn't
pinned, any restart can change the behaviour above, and no offline test can see that.
`filen-s3-probe` checks a live gateway:

```bash
export FILEN_ENDPOINT=https://filen-s3.example.com FILEN_BUCKET=my-bucket \
       FILEN_ACCESS_KEY=... FILEN_SECRET_KEY=...
filen-s3-probe            # exit 1 if a guarantee of this package fails
filen-s3-probe --strict   # also exit 1 if a documented quirk has changed
```

```
  ok       guarantee byte round-trip: 4096 random bytes read back identical
  ok       guarantee file stream round-trip: 3 MiB file streamed up and down identical
  ok       guarantee clean listing: 2 objects, no markers, sizes match
  ok       guarantee exists / delete: exists() true after put, false after delete
  ok       quirk     aws-chunked corrupts uploads: default-checksum upload of 4096 bytes stored as 4140 bytes
  ok       quirk     download_file fails: download_file refused (400 400)
  ok       quirk     ListObjectsV2 requires Prefix: list_objects_v2 without Prefix refused (400 BadRequest)
  ok       quirk     directory markers: raw listing has 2 directory markers
  ok       quirk     HEAD lags behind PUT: HEAD right after PUT answered 401 401, 3s later 200
  ok       quirk     metadata dropped: x-amz-meta-* not returned on GET
  ok       quirk     no multipart upload: create_multipart_upload refused (501 NotImplemented)
  ok       quirk     anonymous GET refused: anonymous GET answered 400 'Required headers are missing'
  ok       quirk     presigned GET refused: presigned GET answered 400 'Required headers are missing'

13 checks: 0 failed, 0 changed
```

It writes only under `_df-s3-filen-probe/<random>/` and deletes everything it created at
the end, markers included. That includes the deliberately corrupted object from check 1.
Run it after upgrading the gateway, or on a schedule.

## Kubernetes deployment

`kubernetes/` is a Helm chart that runs the Filen CLI's S3 gateway itself (`filen s3`) —
the thing this package's `FILEN_ENDPOINT` points at — plus a `Service` and `Ingress` in
front of it.

```bash
helm install filen-s3 ./kubernetes \
  --namespace filen --create-namespace \
  --set filen.email=you@example.com \
  --set filen.password='your-filen-password' \
  --set ingress.host=filen-s3.example.com
```

`filen.email`, `filen.password` and `ingress.host` are required; installation fails with
a clear error if any is missing. The S3 access key and secret are generated by Helm and
stored in a `Secret` (`<release>-filen-s3`) rather than in the chart itself — set
`s3.accessKeyId` / `s3.secretAccessKey` to pin them, or leave them unset to let Helm
generate random ones. `helm upgrade` re-reads whatever is already in the cluster, so
credentials never rotate underneath you. `helm install` prints the `kubectl` command to
read them back:

```
export AWS_ACCESS_KEY_ID=$(kubectl get secret filen-s3-filen-s3 -n filen -o jsonpath="{.data.s3-access-key-id}" | base64 -d)
export AWS_SECRET_ACCESS_KEY=$(kubectl get secret filen-s3-filen-s3 -n filen -o jsonpath="{.data.s3-secret-access-key}" | base64 -d)
```

See `kubernetes/values.yaml` for the rest of the knobs (image, replica count, ingress
class/annotations).

## Development

```bash
uv sync
uv run pytest             # offline; botocore Stubber + request capture, no network
uv run ruff check . && uv run ruff format --check .
```

The tests intercept real botocore requests just before they are sent, so the aws-chunked
fix is checked on the actual request headers rather than assumed from the config.

## Releasing

Publishing is GitHub Actions + PyPI trusted publishing (no API token). A tag whose name
matches the `pyproject.toml` version triggers `.github/workflows/publish.yml`: it runs CI,
builds, smoke-tests the wheel and publishes.

```bash
uv version --bump patch                 # or minor / major
git commit -am "Release $(uv version --short)"
git tag "v$(uv version --short)"
git push --follow-tags
```

A tag that doesn't match the version fails the build before anything is published.

## License

MIT
