Metadata-Version: 2.4
Name: turbofile
Version: 0.1.5
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Rust
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Framework :: AsyncIO
Classifier: Topic :: System :: Filesystems
License-File: LICENSE-MIT
License-File: LICENSE-APACHE
Summary: Real async file I/O for Python: io_uring, IOCP and POSIX AIO via Rust, with an aiofiles-compatible API
Keywords: async,asyncio,file-io,io_uring,aio,iocp,aiofiles,rust
Author: 4thel00z
License-Expression: MIT OR Apache-2.0
Requires-Python: >=3.12
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
Project-URL: Issues, https://github.com/4thel00z/turbofile/issues
Project-URL: Repository, https://github.com/4thel00z/turbofile

<h1 align="center">turbofile</h1>

<p align="center">
  <b>Real async file I/O for Python.</b><br>
  A Rust core drives the best completion mechanism each OS has (io_uring on
  Linux, POSIX AIO on macOS) behind an aiofiles-compatible <code>asyncio</code> API.
</p>

<p align="center">
  <a href="https://github.com/4thel00z/turbofile/actions/workflows/ci.yaml"><img src="https://github.com/4thel00z/turbofile/actions/workflows/ci.yaml/badge.svg?branch=master" alt="CI"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.12%2B-blue" alt="Python 3.12+"></a>
  <a href="https://www.rust-lang.org/"><img src="https://img.shields.io/badge/core-rust-orange" alt="Rust core"></a>
  <a href="#license"><img src="https://img.shields.io/badge/license-MIT%20OR%20Apache--2.0-blue" alt="License: MIT OR Apache-2.0"></a>
</p>

<p align="center">
  <a href="#installation">Installation</a> ·
  <a href="#usage">Usage</a> ·
  <a href="#benchmarks">Benchmarks</a> ·
  <a href="#backends">Backends</a> ·
  <a href="#limitations">Limitations</a> ·
  <a href="#development">Development</a>
</p>

---

Existing Python async-file libraries dispatch every call to a thread pool.
turbofile submits the I/O to the kernel and completes your `await` when the
kernel says the data moved:

- **Zero-copy reads and writes.** The kernel fills the `bytes` object your
  `await f.read(n)` returns; writes pin your buffer and hand the kernel its
  pointer. No intermediate copies on the hot paths.
- **Completion batching.** A burst of completions costs one event-loop wakeup:
  a byte on a pipe the loop watches, so the driver thread never takes the GIL
  to wake the loop, and the batch is drained entirely in Rust.
- **Parallel large reads.** Read-all on a large file is split into 512 KiB
  chunks filled concurrently into one buffer, which spreads the page-cache
  copy across the kernel's I/O threads and beats a single blocking `read`.
- **Whole-file ops.** `read_bytes`/`write_bytes` do open+read/write+close as a
  single submission: one round trip per file. Above 1 MiB, `read_bytes` takes
  the parallel-chunk fill instead, so the kernel writes the returned `bytes`
  directly with no intermediate copy.

## Installation

First release pending; until it's on PyPI, build from source:

```
git clone https://github.com/4thel00z/turbofile
cd turbofile
make develop   # uv + maturin, builds the extension into the venv
```

## Usage

```python
import turbofile

async def main() -> None:
    async with turbofile.open("data.bin", "rb") as f:
        payload = await f.read()

    async with turbofile.open("log.txt", "a", encoding="utf-8") as f:
        await f.write("one line\n")

    data = await turbofile.read_bytes("data.bin")   # whole file, one op
    text = await turbofile.read_text("notes.md")
```

`turbofile.open` mirrors `aiofiles.open`: binary and text modes, buffering,
`encoding`/`errors`/`newline` with full universal-newline semantics,
`seek`/`tell` (text cookies follow CPython's `_pyio` scheme), `readline`,
async iteration, `readinto`, `truncate`, `fsync` via `sync`. Migration is
`import turbofile as aiofiles` for the `open` API.

Inside a running loop, `read_bytes` and `write_bytes` return the completion
future itself rather than a coroutine, so `asyncio.gather` over many files
schedules no task per file. They can still be awaited directly or wrapped in
`asyncio.create_task`; outside a loop they return a coroutine, so
`asyncio.run(turbofile.read_bytes(path))` works too.

`f.read_at(pos, n)` and `f.readinto_at(pos, buffer)` read at a position
without moving the file's own. Inside a running loop they return the
completion future as well, and pages already resident settle it at once, so
`asyncio.gather` over many positions reads them concurrently with no task per
read.

## Benchmarks

`make bench` compares against aiofiles on your machine: p50 per operation,
median of three runs. On an Apple-silicon Mac (macOS 26.4, POSIX AIO backend,
page-cache-hot files, mains power):

| workload                                   | vs aiofiles |
| ------------------------------------------ | ----------- |
| 4 KiB read on an open file                 | 59x         |
| 32 concurrent 4 KiB random reads           | 55x         |
| 200 small files read concurrently          | 14.0x       |
| 4 KiB whole-file read (`read_bytes`)       | 3.5x        |
| 8 MiB whole-file read (`read_bytes`)       | 1.4x        |
| 8 MiB whole-file read (`open` + `read`)    | 1.5x        |
| 8 MiB sequential write (1 MiB chunks)      | 1.0x        |

Both 8 MiB reads finish in about 0.31 ms, against 0.27 ms for a blocking
`read` of the same bytes and 0.44 ms for aiofiles: at that size the copy is
the floor and aiofiles pays its thread hop on top of it. The sixteen chunks
copying in parallel on the kernel's AIO threads
show at 64 MiB, where the parallel fill takes 2.7 ms against 8.9 ms for a read
in a thread. Sequential writes await one chunk at a time, so one copy is in
flight and the write row sits at the copy as well; turbofile wins where per-op
overhead and concurrency dominate, which is what an asyncio application does.

Reads whose pages are already resident never leave the event-loop thread: no
submission, no driver-thread hop, no completion wakeup. On Linux the kernel
decides, through `preadv2(RWF_NOWAIT)`; on macOS `mincore` on a mapping of the
file reports which pages the buffer cache holds, and a plain `pread` copies
them. Data that is not resident takes the async path. The table's 32
concurrent random reads take this route through `read_at`. `make ladder`
isolates the read: its `file_read` rung is `BinaryFile`'s positional read of a
hot 4 KiB, its `pread` rung the blocking syscall for the same bytes.

| platform    | `file_read` | `pread` |
| ----------- | ----------- | ------- |
| Linux, ext4 | 1.3 us      | 1.1 us  |
| macOS, APFS | 1.2 us      | 0.5 us  |

On Linux `read_bytes` takes the same inline route for the whole file, open
included, through `openat2(RESOLVE_CACHED)`: 5.3 us for a hot 4 KiB file. macOS
has no cached-only `open`, and an inline `open` there stalls on cold paths and
endpoint-security scans (about 300 us for the first open of a file on a machine
running Jamf Protect), so `read_bytes` keeps its open and close on the driver
thread.

Benchmark on a real filesystem: tmpfs sets no `FMODE_NOWAIT`, which disables
the fast path and makes io_uring punt every read to a kernel worker, so `/tmp`
numbers are not comparable. `make bench` and `make ladder` take `FS=<dir>` and
print which filesystem they measured. See `perf/README.md` for the analysis
harness.

## Backends

| OS      | backend                | mechanism                                    |
| ------- | ---------------------- | -------------------------------------------- |
| Linux   | compio (fusion driver) | io_uring, automatic polling fallback under seccomp/old kernels |
| macOS   | darwin-aio             | POSIX AIO (`aio_read`/`aio_write`/`aio_fsync` in XNU) |
| macOS   | compio (opt-in)        | kqueue polling driver with thread dispatch   |

`TURBOFILE_BACKEND=compio` selects the compio driver on macOS (benchmarking,
or as an escape hatch). Windows (IOCP via compio) is planned.

## Limitations

- `opener=` and integer file descriptors are not supported.
- Cancelling an `await` sends a kernel-level abort request (`aio_cancel` on
  macOS; the compio driver stops before the next chunk), best-effort by
  nature: an op the kernel finishes first settles with its result. Either way
  the `CancelledError` arrives only once the op has settled, so your buffer
  is never touched after the `await` raises.

## Development

```
make develop   # build the extension into the venv (uv + maturin)
make test      # cargo test + pytest
make lint      # clippy -D warnings
make bench     # benchmark against aiofiles
```

## License

MIT OR Apache-2.0

