Metadata-Version: 2.4
Name: opendalfs
Version: 0.0.3
License: Apache-2.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <4.0,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fsspec
Requires-Dist: opendal<0.48,>=0.47.5
Provides-Extra: dev
Requires-Dist: ruff; extra == "dev"
Provides-Extra: test
Requires-Dist: pytest; extra == "test"
Requires-Dist: pytest-asyncio; extra == "test"
Requires-Dist: pytest-cov; extra == "test"
Requires-Dist: pyarrow; extra == "test"
Requires-Dist: s3fs; extra == "test"
Requires-Dist: boto3; extra == "test"
Provides-Extra: bench
Requires-Dist: pyarrow; extra == "bench"
Requires-Dist: s3fs; extra == "bench"
Requires-Dist: boto3; extra == "bench"
Provides-Extra: all
Requires-Dist: opendalfs[bench,dev,test]; extra == "all"
Dynamic: license-file

# opendalfs

[![Discussions](https://img.shields.io/github/discussions/fsspec/opendalfs)](https://github.com/fsspec/opendalfs/discussions)
[![Tests](https://github.com/fsspec/opendalfs/actions/workflows/tests.yml/badge.svg)](https://github.com/fsspec/opendalfs/actions/workflows/tests.yml)


`opendalfs` is a Python-based interface for file systems that enables interaction with different storage services by [Apache OpenDAL](https://github.com/apache/opendal). Through `opendalfs`, users can utilize fsspec's standard API to operate on all [storage services supported by OpenDAL](https://docs.rs/opendal/latest/opendal/services/index.html).

## URL Protocols

`opendalfs` registers multiple fsspec protocols in the form of `opendal+<service>`, for example:

```python
import fsspec

f = fsspec.open(
    "opendal+s3://my-bucket/path/to/file",
    mode="rb",
    endpoint="http://localhost:9000",
    access_key_id="minioadmin",
    secret_access_key="minioadmin",
)
```

The URL host is mapped to the service container (e.g. `bucket` for `s3`/`gcs`, `container` for `azblob`), and the URL path is used as the object key.

For other OpenDAL services, register protocols at runtime:

```python
import opendalfs

opendalfs.register_opendal_service("oss")
```

## Installation

### Basic Installation

```bash
pip install opendalfs
```

### Development Installation

```bash
# Install all development dependencies
pip install "opendalfs[all]"

# Or install specific groups
pip install "opendalfs[dev,test]"  # for development and testing
```

## Development Setup

This project uses:
- Python 3.12+ for the Python interface
- ruff for code formatting and linting
- pytest for testing

For development setup and guidelines, see our [Contributing Guide](https://github.com/fsspec/opendalfs/blob/main/CONTRIBUTING.md).

## Benchmarks

The benchmark script compares Arrow direct, opendalfs (fsspec), and s3fs (fsspec) on MinIO.

```bash
uv sync --extra bench
uv run python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4
```

Configure MinIO access via `OPENDAL_S3_ENDPOINT`, `OPENDAL_S3_BUCKET`,
`OPENDAL_S3_ACCESS_KEY_ID`, and `OPENDAL_S3_SECRET_ACCESS_KEY`.

You can run MinIO locally with the official binary, for example:

```bash
./minio server ./minio-data --console-address ":9001"
```

For profiling, you can install a tool with `uv` (for example `py-spy`) and run:

```bash
uv tool install py-spy
uv tool run py-spy record -o bench.svg -- python bench/bench_read_write.py --sizes 16,32,64 --files 4 --workers 4
```

High write concurrency can stall on some systems. If runs time out, reduce
`--fsspec-workers`.

## Status

See [Tracking issues of 0.1.0 version for opendalfs](https://github.com/fsspec/opendalfs/issues/6)

## Contributing

opendalfs is an exciting project currently under active development. Whether you're looking to use it in your projects or contribute to its growth, there are several ways you can get involved:

- Follow the [Contributing Guide](https://github.com/fsspec/opendalfs/blob/main/CONTRIBUTING.md) to contribute
- Create new [Issue](https://github.com/fsspec/opendalfs/issues/new) for bug reports or feature requests
- Join discussions in [Discussions](https://github.com/fsspec/opendalfs/discussions)

## Getting Help

- Submit [issues](https://github.com/fsspec/opendalfs/issues/new/choose) for bug reports
- Ask questions in [discussions](https://github.com/fsspec/opendalfs/discussions/new?category=q-a)

## License

Licensed under [Apache License, Version 2.0](https://github.com/fsspec/opendalfs/blob/main/LICENSE).
