Metadata-Version: 2.4
Name: bucket-helper
Version: 0.2.2
Summary: Bucket Helper — utility functions for AWS S3 and any S3-compatible object storage (MinIO, Backblaze B2, DigitalOcean Spaces, Cloudflare R2, Wasabi, etc.) via boto3, exposed as library, argparse CLI, click CLI, FastAPI HTTP surface and MCP tools. credentials() loader, upload / download / list / delete / exists / make_bucket / remote_tempfile.
Author-email: Warith HARCHAOUI <warithmetics@deraison.ai>
License: BSD-3-Clause
Project-URL: Homepage, https://github.com/warith-harchaoui/bucket-helper
Project-URL: Issues, https://github.com/warith-harchaoui/bucket-helper/issues
Keywords: s3,minio,boto3,object-storage,aws,r2,b2,spaces,bucket,cli,fastapi,mcp
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Internet :: File Transfer Protocol (FTP)
Classifier: Topic :: System :: Archiving
Requires-Python: <3.14,>=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: os-helper>=1.4.2
Requires-Dist: boto3<2,>=1.34
Requires-Dist: botocore<2,>=1.34
Provides-Extra: cli
Requires-Dist: click<9,>=8.1; extra == "cli"
Provides-Extra: api
Requires-Dist: fastapi<1,>=0.111; extra == "api"
Requires-Dist: uvicorn<1,>=0.30; extra == "api"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "api"
Provides-Extra: mcp
Requires-Dist: fastapi-mcp<1,>=0.3; extra == "mcp"
Requires-Dist: fastapi<1,>=0.111; extra == "mcp"
Requires-Dist: uvicorn<1,>=0.30; extra == "mcp"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "mcp"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pyyaml>=6; extra == "dev"
Requires-Dist: moto[s3]>=5; extra == "dev"
Requires-Dist: httpx>=0.27; extra == "dev"
Requires-Dist: click<9,>=8.1; extra == "dev"
Requires-Dist: fastapi<1,>=0.111; extra == "dev"
Requires-Dist: python-multipart<1,>=0.0.9; extra == "dev"
Requires-Dist: fastapi-mcp<1,>=0.3; extra == "dev"
Dynamic: license-file

# Bucket Helper

[🇫🇷](https://github.com/warith-harchaoui/bucket-helper/blob/main/LISEZMOI.md) · [🇬🇧](https://github.com/warith-harchaoui/bucket-helper/blob/main/README.md)

[![CI](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml/badge.svg)](https://github.com/warith-harchaoui/bucket-helper/actions/workflows/ci.yml) [![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://github.com/warith-harchaoui/bucket-helper/blob/main/LICENSE) [![Python](https://img.shields.io/badge/python-3.10%E2%80%933.13-blue.svg)](#)

`Bucket Helper` belongs to a collection of libraries called `AI Helpers` developed for building Artificial Intelligence.

Utility functions for **AWS S3** and any **S3-compatible object storage** — MinIO, Backblaze B2 S3 API, DigitalOcean Spaces, Cloudflare R2, Wasabi, and friends. Built on [boto3](https://boto3.amazonaws.com/v1/documentation/api/latest/index.html). Same shape as [sftp-helper](https://github.com/warith-harchaoui/sftp-helper): a `credentials()` loader, the usual CRUD (`upload` / `download` / `delete` / `exists` / `list_prefix`), and a `remote_tempfile` context manager for stage-and-share flows.

[🌍 AI Helpers](https://harchaoui.org/warith/ai-helpers)

[![logo](https://raw.githubusercontent.com/warith-harchaoui/bucket-helper/main/assets/logo.png)](https://harchaoui.org/warith/ai-helpers)

# Installation

**Prerequisites** — **Python 3.10–3.13** and **git**, cross-platform:

- 🍎 **macOS** ([Homebrew](https://brew.sh)): `brew install python git`
- 🐧 **Ubuntu/Debian**: `sudo apt update && sudo apt install -y python3 python3-pip git`
- 🪟 **Windows** (PowerShell): `winget install Python.Python.3.12 Git.Git`

Then install the package:


```bash
pip install --force-reinstall --no-cache-dir git+https://github.com/warith-harchaoui/bucket-helper.git@v0.2.2
```

Optional extras — pick what you need:

```bash
# argparse CLI is always available. Add the click twin:
pip install 'bucket-helper[cli] @ git+https://github.com/warith-harchaoui/bucket-helper.git@v0.2.2'

# HTTP server (FastAPI + uvicorn + python-multipart):
pip install 'bucket-helper[api] @ git+https://github.com/warith-harchaoui/bucket-helper.git@v0.2.2'

# MCP tools (fastapi-mcp) — requires the [api] plumbing:
pip install 'bucket-helper[api,mcp] @ git+https://github.com/warith-harchaoui/bucket-helper.git@v0.2.2'
```

# Configuration

A ready-to-fill template is committed at [`s3_config.json.example`](https://github.com/warith-harchaoui/bucket-helper/blob/main/s3_config.json.example). Copy it to `s3_config.json` and edit in place — real `*config.json` files are gitignored so you cannot accidentally commit secrets:

```bash
cp s3_config.json.example s3_config.json
# then edit s3_config.json with your AWS / MinIO / R2 / B2 credentials
```

You may also write a `s3_config.yaml`, use a `.env`, or set environment variables — `bucket-helper` falls back in that order via `os_helper.get_config`. Required keys:

```json
{
  "s3_access_key": "AKIA...",
  "s3_secret_key": "...",
  "s3_bucket":     "my-bucket",
  "s3_https":      "https://my-bucket.s3.eu-west-3.amazonaws.com"
}
```

Optional keys:

| Key | Default | Notes |
|---|---|---|
| `s3_region` | `"us-east-1"` | AWS region; mostly cosmetic for MinIO / R2 |
| `s3_endpoint_url` | empty (= AWS S3) | Set this for S3-compatible backends — see table below |
| `s3_prefix` | empty | Default key prefix added by `upload(...)` when no destination is given |
| `s3_use_path_style` | `"false"` | Force path-style addressing (`endpoint/bucket/key` instead of `bucket.endpoint/key`). Typical for MinIO with custom domains. |
| `s3_verify_ssl` | `"true"` | Disable only for dev MinIO with self-signed certs |

## Endpoint URLs for common S3-compatible storage

Set `s3_endpoint_url` to:

| Provider | Endpoint |
|---|---|
| **AWS S3** | leave empty / unset |
| **MinIO** | `http://minio.example.com:9000` (or `https://...` with TLS) |
| **DigitalOcean Spaces** | `https://nyc3.digitaloceanspaces.com` (region in subdomain) |
| **Cloudflare R2** | `https://<account_id>.r2.cloudflarestorage.com` |
| **Backblaze B2 (S3 API)** | `https://s3.<region>.backblazeb2.com` |
| **Wasabi** | `https://s3.<region>.wasabisys.com` |

# Usage

For the full catalog of recipes (uploads / downloads / listings, S3-compatible endpoints — MinIO / R2 / B2 / Spaces / Wasabi, temporary remote keys with auto-cleanup, mirroring with sftp-helper), see [📋 EXAMPLES.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/EXAMPLES.md).

```python
import bucket_helper as bh

# Load creds — JSON / YAML / env / .env (auto-fallback in that order)
cred = bh.credentials("path/to/s3_config.json")

# Upload a local file
uri = bh.upload("local.txt", cred, "folder/uploaded.txt")
# uri == "s3://my-bucket/folder/uploaded.txt"

assert bh.exists(uri, cred)

# Download
bh.download(uri, "downloaded.txt", cred)

# List
for key in bh.list_prefix("folder/", cred):
    print(key)

# Delete
bh.delete(uri, cred)
```

## MinIO example

```python
cred = {
    "s3_access_key":      "minioadmin",
    "s3_secret_key":      "minioadmin",
    "s3_bucket":          "uploads",
    "s3_https":           "http://minio.example.com:9000/uploads",
    "s3_endpoint_url":    "http://minio.example.com:9000",
    "s3_use_path_style":  "true",
    "s3_region":          "us-east-1",  # MinIO accepts any region string
}

bh.make_bucket("uploads", cred)
bh.upload("file.bin", cred, "file.bin")
```

## Stage-and-share with `remote_tempfile`

Drop a generated file at a unique random key, hand the public URL to a
downstream worker / webhook, and the object is deleted on block exit
(even if the body raises):

```python
import bucket_helper as bh
import requests

cred = bh.credentials("path/to/s3_config.json")

with bh.remote_tempfile(cred, ext="json", prefix="runs") as (s3_addr, public_url):
    bh.upload("payload.json", cred, s3_addr, content_type="application/json")
    # Hand the URL to something that fetches it once.
    requests.post("https://hook.example.com/process", json={"input_url": public_url}).raise_for_status()
# Object is gone here, no manual cleanup.
```

# Multi-surface exposure

Every public function in the library is also exposed as:

- **argparse CLI** — `bucket-helper <subcommand>` (installed by default).
- **click CLI** — `bucket-helper-click <subcommand>` (install `[cli]` extra).
- **FastAPI HTTP** — `uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000` (install `[api]` extra).
- **MCP tools** — `bucket-helper-mcp` (install `[api,mcp]` extras).

Both CLIs share the same subcommand names and flags — pick your favourite.

## CLI examples

```bash
# argparse CLI (always available)
bucket-helper upload      --config s3_config.json --input local.txt --key folder/uploaded.txt
bucket-helper exists      --config s3_config.json --key folder/uploaded.txt
bucket-helper download    --config s3_config.json --key folder/uploaded.txt --output back.txt
bucket-helper list        --config s3_config.json --prefix folder/
bucket-helper delete      --config s3_config.json --key folder/uploaded.txt
bucket-helper make-bucket --config s3_config.json --bucket new-bucket
bucket-helper tempfile    --config s3_config.json --ext json --prefix runs
bucket-helper strip-path  --config s3_config.json --address s3://my-bucket/path/to/obj

# click CLI — same verbs, same flags
bucket-helper-click upload --config s3_config.json --input local.txt --key folder/uploaded.txt
```

## HTTP + MCP server

```bash
# Serve HTTP + MCP (default credentials picked up from BUCKET_HELPER_CONFIG)
BUCKET_HELPER_CONFIG=$PWD/s3_config.json bucket-helper-mcp

# Or run only FastAPI directly:
uvicorn bucket_helper.api:app --host 0.0.0.0 --port 8000
# → Swagger UI at http://localhost:8000/docs
```

Per-request credentials can also be sent as multipart form fields
(`s3_access_key` / `s3_secret_key` / `s3_bucket` / `s3_https` / …).

## Docker

```bash
docker build -t bucket-helper .
docker run --rm -p 8000:8000 \
  -e BUCKET_HELPER_CONFIG=/config/s3_config.json \
  -v $PWD/s3_config.json:/config/s3_config.json:ro \
  bucket-helper
```

See also: [LANDSCAPE.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/LANDSCAPE.md) (competitive positioning) and
[GUI.md](https://github.com/warith-harchaoui/bucket-helper/blob/main/GUI.md) (visual product design plan).

# Author
 - [Warith HARCHAOUI](https://linkedin.com/in/warith-harchaoui)

# Acknowledgements
Special thanks to [Mohamed Chelali](https://mchelali.github.io) and [Bachir Zerroug](https://www.linkedin.com/in/bachirzerroug) for fruitful discussions.
