Metadata-Version: 2.4
Name: alocals3
Version: 0.7.0
Classifier: Development Status :: 3 - Alpha
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
License-File: LICENSE
Summary: alocals3: a Rust S3-like server and Rust-backed Python client
Keywords: s3,rust,local
Author: alocals3 contributors
License-Expression: MIT
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Homepage, https://github.com/sxysxy/alocals3
Project-URL: Repository, https://github.com/sxysxy/alocals3

[简体中文](README.zh-CN.md)

# alocals3

`alocals3` is a local S3-like object store focused on fast local development and internal workloads.

The current `main` branch is Rust-first:

- Server: pure Rust binary, no Python runtime required.
- Metadata backend: SQLite or PostgreSQL.
- Object payloads: local filesystem with SHA-256 based sharding.
- Python client: wheel package backed by Rust networking through `reqwest`.
- Python target: Python 3.12+ with PyO3 `abi3-py312` limited API.

The project implements an S3-compatible subset, not the full AWS S3 API surface.

## Quick Start

Build and run the Rust server:

```bash
PYO3_NO_PYTHON=1 cargo build --release --no-default-features --features server,server-binary --bin alocals3-server

target/release/alocals3-server \
  --host 127.0.0.1 \
  --port 8000 \
  --database-url "sqlite:///./alocals3.db" \
  --storage-root ./data
```

Background garbage collection is enabled by default. It periodically removes object files that are no longer referenced by the database and removes database records whose files are missing. Candidates must be older than the GC grace period before they are deleted.

```bash
target/release/alocals3-server \
  --gc-interval-secs 300 \
  --gc-grace-secs 300 \
  --gc-start-delay-secs 30
```

Set `--disable-gc` to disable background GC. `--gc-interval-secs 0` and `ALOCALS3_GC_INTERVAL_SECS=0` are equivalent.

Run the reproducible GC correctness suite:

```bash
./examples/gc_correctness.sh
```

It covers orphan files, missing-file records, stale temp files, live/shared object retention, grace-period behavior, GC disable modes, and orphan-path reuse races. Latest local result:

```text
GC correctness and race scenarios passed
orphan_files_deleted=1 missing_records_deleted=1 stale_tmp_files_deleted=1
remaining_objects: grace=1 disable_flag=1 interval_zero=1 race=30
```

Use PostgreSQL instead of SQLite:

```bash
target/release/alocals3-server \
  --host 127.0.0.1 \
  --port 8000 \
  --database-url "postgresql://user:password@127.0.0.1:5432/alocals3" \
  --storage-root ./data
```

Install the Python client from source:

```bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip maturin
python -m pip install -e .
```

## Build Artifacts

Release helper scripts live in [scripts](scripts/README.md):

```bash
# Linux static server plus Python 3.12+ ABI3 wheel
scripts/build-linux-release.sh

# macOS arm64, macOS 11 deployment baseline
scripts/build-macos-release.sh

# Windows 10+ PowerShell
.\scripts\build-windows-release.ps1
```

Linux server builds default to `x86_64-unknown-linux-musl`; Linux wheels default to `manylinux_2_28`. macOS builds default to `aarch64-apple-darwin` with `MACOSX_DEPLOYMENT_TARGET=11.0`.

The wheel is configured as Python 3.12+ ABI3 via PyO3 `abi3-py312`. It is not a `cp312-cp312` wheel unless the ABI3 feature is removed.

Platform wheels include the Rust server executable and install an `alocals3-server` command. Standalone server artifacts are also named `alocals3-server` on Unix-like platforms and `alocals3-server.exe` on Windows.

## Configuration

Server CLI flags:

- `--host`: bind host, default `127.0.0.1`
- `--port`: bind port, default `8000`
- `--database-url`: SQLite or PostgreSQL URL
- `--storage-root`: object payload root directory
- `--log-level`: log level or `tracing_subscriber` filter directive, default `info`
- `--max-upload-size`: maximum upload body size in bytes, default `0` disables the axum body limit
- `--version`: print server version and author information

Environment variables:

- `ALOCALS3_DATABASE_URL`: default `sqlite:///./alocals3.db`
- `ALOCALS3_STORAGE_ROOT`: default `./data`
- `ALOCALS3_LOG_LEVEL`: default `info`
- `ALOCALS3_MAX_UPLOAD_SIZE`: default `0`

Database URL examples:

- SQLite: `sqlite:///./alocals3.db`
- PostgreSQL: `postgresql://user:password@127.0.0.1:5432/alocals3`

Use an absolute SQLite path in scripts and services to avoid accidentally writing to different database files from different working directories. PostgreSQL is recommended for sustained concurrent workloads.

Logging is written to stderr. Common levels are `debug`, `info`, `warn`, and `error`; full `EnvFilter` directives such as `warn,alocals3_server=debug` are also accepted.

Uploads are not capped by the server body extractor by default. The current implementation buffers each uploaded object in memory before writing it to disk, so practical limits are still determined by process memory and available storage. Set `--max-upload-size` to restore an explicit byte limit.

## Storage Layout

- Bucket and object metadata is stored in SQLite or PostgreSQL.
- Object bytes are stored on local disk.
- Blob paths are content-addressed and sharded:
  - `sha256(<object bytes>) = <digest>`
  - `{storage_root}/objects/{digest[:2]}/{digest[2:4]}/{digest}`

Object keys, bucket names, prefixes, delimiters, and continuation tokens are UTF-8 text. Client path parameters are UTF-8 percent-encoded automatically; pass raw strings such as `logs/data.txt` or `logs/数据.txt`, not pre-encoded URL fragments.

## HTTP API

- `GET /healthz`: health check
- `GET /s3`: list buckets
- `PUT /s3/{bucket}`: create bucket
- `DELETE /s3/{bucket}`: delete empty bucket
- `GET /s3/{bucket}/objects`: list objects
- `GET /s3/{bucket}?list-type=2`: S3-style ListObjectsV2
- `PUT /s3/{bucket}/{key}`: upload object
- `GET /s3/{bucket}/{key}`: download object
- `HEAD /s3/{bucket}/{key}`: object metadata
- `DELETE /s3/{bucket}/{key}`: delete object

Supported object features:

- `ETag` is the MD5 hex digest of the object body.
- `Range` requests return `206` or `416`.
- `If-None-Match` and `If-Match` are supported for `PUT`.
- `Content-MD5` is validated on `PUT`.
- `If-None-Match` is supported for `GET` and `HEAD`.

`PUT /s3/{bucket}/{key}` returns:

- `201`: new object created
- `200`: existing object overwritten
- `400`: invalid `Content-MD5`
- `412`: conditional request failed

## Client Usage

The Python runtime dependency list is intentionally empty. HTTP networking is implemented in Rust, not `httpx`.

```python
import asyncio
from pathlib import Path

from alocals3.client import LocalS3Client, LocalS3ClientAsync

with LocalS3Client("http://127.0.0.1:8000", disable_proxy=True) as client:
    client.create_bucket("demo")
    info = client.put_object("demo", "logs/数据.txt", Path("data.txt"))
    print(info["etag"])

    data, headers = client.get_object_range("demo", "logs/数据.txt", "bytes=0-99")
    print(len(data), headers.get("content-range"))

    with client.open("s3://demo/logs/数据.txt", "r") as f:
        print(f.read())

    with client.open("s3://demo/logs/from-open.txt", "wb") as f:
        f.write(b"hello from file-like API\n")

    client.get_object_to_file("demo", "logs/数据.txt", Path("copy.txt"))


async def main() -> None:
    async with LocalS3ClientAsync("http://127.0.0.1:8000", disable_proxy=True) as client:
        print(await client.list_buckets())


asyncio.run(main())
```

CLI:

```bash
python -m alocals3.client --endpoint http://127.0.0.1:8000 CREATE_BUCKET demo
python -m alocals3.client --endpoint http://127.0.0.1:8000 PUT demo file.bin ./file.bin
python -m alocals3.client --endpoint http://127.0.0.1:8000 GET demo file.bin ./copy.bin
python -m alocals3.client --endpoint http://127.0.0.1:8000 LIST_OBJECTS_V2 demo --prefix logs/ --delimiter /
```

Set `disable_proxy=True` or pass `--disable-proxy` to ignore proxy environment variables such as `HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`, and `NO_PROXY`.

`client.open()` is file-like but not an in-memory object wrapper:

- Read modes (`"rb"` / `"r"`) create a Rust-backed streaming HTTP reader. `open()` sends the request and reads response headers, but object bytes are pulled from the network when the returned file object's `read()` path runs.
- Write modes (`"wb"` / `"w"`) spool writes to a Rust-owned temporary file. The HTTP `PUT` is sent when the file is closed or the `with` block exits successfully. If the `with` block exits with an exception, the upload is discarded.
- `cache_path=` is best-effort and is populated as bytes pass through the file-like object; cache write failures do not fail the network operation.

Benchmark the Python file-like API with a large object:

```bash
./examples/benchmark_file_like.sh
```

Validate stream multiplexing and Range-based resume reads:

```bash
./examples/benchmark_stream_multiplex.sh
```

The current file-like API supports resume reads through `range_header="bytes=N-"`. Upload resume is not implemented yet; write modes spool to a Rust-owned temporary file and use one HTTP `PUT` on close.

## Curl Examples

```bash
curl -i -X PUT http://127.0.0.1:8000/s3/demo
curl -i -X PUT --data-binary @file.bin http://127.0.0.1:8000/s3/demo/file.bin
curl -i http://127.0.0.1:8000/s3/demo/file.bin
curl -i -H "Range: bytes=0-99" http://127.0.0.1:8000/s3/demo/file.bin
curl -sS "http://127.0.0.1:8000/s3/demo?list-type=2&prefix=logs/&delimiter=/&max-keys=100"
```

Conditional PUT:

```bash
curl -i -X PUT -H "If-None-Match: *" --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin

curl -i -X PUT -H 'If-Match: "d41d8cd98f00b204e9800998ecf8427e"' --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin

MD5_B64=$(openssl md5 -binary file.bin | openssl base64)
curl -i -X PUT -H "Content-MD5: ${MD5_B64}" --data-binary @file.bin \
  http://127.0.0.1:8000/s3/demo/file.bin
```

## Consistency Notes

- Object bytes are written through a temporary file and atomic rename.
- Metadata updates are committed through the selected database backend.
- This is not a single distributed transaction across database and filesystem.
- Under process or machine failure, orphan blob files may exist. The Python package no longer ships an alternate storage backend or Python server path; operational cleanup should be handled outside the request path.

## Updates

[updates.md](updates.md)

## License

[The MIT License](LICENSE)

