Metadata-Version: 2.5
Name: filetx
Version: 0.1.0
Summary: Transactional filesystem operations - batch moves, copies and deletes that either all commit or all roll back.
Project-URL: Homepage, https://github.com/Prithv122/filetx
Project-URL: Source, https://github.com/Prithv122/filetx
Project-URL: Issues, https://github.com/Prithv122/filetx/issues
Project-URL: Changelog, https://github.com/Prithv122/filetx/blob/main/CHANGELOG.md
Author-email: Prithvi M <feverishpetroleum16@gmail.com>
License-Expression: MIT
License-File: LICENSE
Keywords: atomic,files,filesystem,rollback,transaction,undo
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: System :: Filesystems
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown

# filetx

> Transactional filesystem operations — batch moves, copies and deletes that either all commit or all roll back.

[![CI](https://github.com/Prithv122/filetx/actions/workflows/ci.yml/badge.svg)](https://github.com/Prithv122/filetx/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/filetx.svg)](https://pypi.org/project/filetx/)
[![Python](https://img.shields.io/pypi/pyversions/filetx.svg)](https://pypi.org/project/filetx/)

**Stack:** Python 3.10+, standard library only, zero runtime dependencies.

```bash
pip install filetx
```

```python
from filetx import Transaction

with Transaction(journal="release.jsonl") as tx:
    tx.mkdir("dist", exist_ok=True)
    tx.move("build/app.bin", "dist/app.bin")
    tx.copy("build/manifest.json", "dist/manifest.json")
    tx.delete("build", missing_ok=True)
```

Nothing happens until the block exits. If any step fails, every completed step
is undone. If the process is killed part-way through, `filetx undo release.jsonl`
finishes the rollback from the journal it left behind.

---

## 1. The problem

Multi-step filesystem work has no rollback. A release script that creates a
directory, moves three artefacts into it and deletes the build tree has four
chances to fail, and if it fails on the third the disk is left in a state that
matches neither the before nor the after. The usual responses are to copy
everything to a backup directory first — which costs time proportional to the
data, so nobody does it for large trees — or to write bespoke cleanup code per
script, which is only exercised on the day it is needed and is therefore
usually wrong.

Python has `shutil` for individual operations and `atomicwrites` for making a
single file write atomic. Neither gives you a *batch* that is all-or-nothing.
`filetx` is that missing piece: record the operations, commit them, and get the
tree back unchanged if anything goes wrong — including if the process dies.

This started as the transaction log inside a file-organiser tool I built, where
a half-finished reorganisation of someone's photo library is a genuinely bad
outcome. The engine turned out to be the interesting part, so it was extracted,
generalised and published.

## 2. What it operates on

There is no dataset here — this is a library, and its input is whatever tree the
caller points it at.

| | |
|---|---|
| Input | Arbitrary files, directories and symlinks on a local filesystem |
| Scale tested | Trees up to 5,000 files; single renames are size-independent |
| Benchmark data | Synthetic, generated by `scripts/benchmark.py` (seed 42), not committed |
| Package licence | MIT |
| Dependencies | None at runtime. `pytest`, `pytest-cov` and `ruff` for development |

**Non-goals.** Not a distributed transaction manager, not crash-safe against
media failure, and not safe against another process mutating the same paths
concurrently — see [Limitations](#limitations).

## 3. Architecture

```mermaid
flowchart TD
    A["tx.move / copy / delete / mkdir<br/>(recorded, not performed)"] --> B["commit()"]
    B --> C["plan_undo — validate,<br/>compute undo record"]
    C -->|invalid| R
    C --> D["journal — write undo record, fsync"]
    D --> E["apply — change the filesystem"]
    E -->|OSError| R
    E --> F{"more operations?"}
    F -->|yes| C
    F -->|no| G["journal — commit"]
    G --> H["purge staged material"]
    R["revert applied operations,<br/>in reverse order"] --> S{"all reverted?"}
    S -->|yes| T["journal — rollback<br/>raise TransactionError"]
    S -->|no| U["journal left unsettled<br/>raise RollbackError"]
    U -.->|"later: filetx undo"| R
```

The undo record is written to the journal **before** the change it describes is
attempted. That is the whole design: a record written afterwards is useless to a
process that died in between.

Writing it first requires knowing, in advance, where a file is about to be moved
to — so staged names are derived deterministically from `(path, transaction id,
plan index)` rather than allocated at random.

The remaining ambiguity is that the journal can describe a change that never
actually happened. Every `revert` therefore inspects the real filesystem before
acting instead of trusting the record, which makes reverting idempotent: running
it twice, or against an operation that never applied, does nothing.

## 4. Key decisions & tradeoffs

| Decision | Chose | Over | Why |
|---|---|---|---|
| Making deletes reversible | Rename the target to a hidden sibling in its own parent directory | Copy it to a staging/backup directory | A sibling is guaranteed to be on the same filesystem, so it is one atomic rename regardless of size — **1,280× faster** on a 5,000-file tree (§5). It also nests correctly: deleting `a/b.txt` then `a` stages the file inside `a`, then renames `a` wholesale, and reverse-order rollback restores `a` before looking for `b.txt` inside it. |
| When to compute the undo record | Before applying (write-ahead) | After applying, returning what was done | The "after" version cannot recover a process killed between the change and the log write — precisely the window that matters. Cost: staged names must be deterministic, and the journal may record changes that never happened, which is why every revert is state-checking. |
| Operations execute | Deferred — recorded, applied on `commit()` | Eagerly, as each method is called | Makes the plan inspectable and gives a dry run (`tx.describe()`) for free, and an exception in the caller's own code inside the `with` block costs nothing to undo. |
| Failure signalling | `TransactionError` and `RollbackError` as distinct types | One exception for "the commit failed" | "Your change didn't happen and the tree is fine" and "your change didn't happen and the tree is in an unknown state" demand completely different responses. Collapsing them would be the most dangerous thing this library could do. |
| Recovery vs. leftover staged data | Report it, never delete it | Clean up automatically | If a revert declined to restore something because the original path is occupied again, the staged copy may be the only copy left. Unattended deletion there is the one unrecoverable mistake available; `filetx undo` prints the paths and stops. |
| Cross-filesystem moves | Detect `EXDEV`, degrade to copy-then-stage | Refuse, or always copy | Keeps the fast path fast and the slow path correct. Documented as O(size) rather than O(1). Any other `OSError` propagates — a permission error must not be silently treated as a volume boundary. |
| Journal format | JSON Lines | SQLite, or a binary log | An operator looking at a half-finished batch can read it with `cat`. Appending one short line is atomic enough at this size, and a torn final line is detected and tolerated. |

## 5. Results

Measured on Windows 11 (NTFS), Python 3.13.9, 12-core CPU / 64 GB RAM, median of
3 runs. Reproduce with `uv run python scripts/benchmark.py --files 5000 --size 1024`.

| Metric | Value | Notes |
|---|---|---|
| Make a 5,000-file delete reversible, then undo it | **7.6 ms** | Two renames |
| Same outcome via copy-to-backup then restore | **9,692 ms** | `shutil.copytree` + `rmtree` + `copytree` |
| Speedup on the reversibility path | **~1,280×** | Widens with tree size; staging is size-independent |
| Commit the same delete (data actually removed) | 1,407 ms | Staging is O(1), *deleting* is not — stated so the number above is not mistaken for magic |
| Tests | **73 passing, 100% line coverage** | Includes two tests that `os._exit()` a real subprocess mid-commit, on either side of the rename, then recover from the journal |
| Python versions | 3.10, 3.11, 3.12, 3.13 | All four run in CI, not just claimed in metadata |
| Platforms | Linux, Windows, macOS | Windows and macOS on 3.13; this library is mostly `os.rename`, which is where platforms disagree |
| Runtime dependencies | 0 | |

The benchmark tree is synthetic and generated by the script above. The
comparison is against `shutil`, i.e. what you would write by hand, not against
another library.

## 6. How to run

```bash
git clone https://github.com/Prithv122/filetx.git
cd filetx
uv sync
uv run pytest --cov=src --cov-report=term-missing
```

No environment variables, services or datasets are needed. Other useful commands:

```bash
uv run python scripts/benchmark.py     # reproduce the numbers in section 5
uv run ruff check . && uv run ruff format --check .
uv build && uv run --with twine twine check --strict dist/*
```

### Recovering an interrupted run

```bash
filetx inspect release.jsonl   # what was this transaction doing, and how far did it get?
filetx undo release.jsonl      # put it back
```

`inspect` marks each operation `done`, `PARTIAL` or `-`. `undo` rolls an
unsettled transaction back, cleans up after a committed one, and reports — but
never deletes — staged data whose original path has since been reoccupied.

### Limitations

- **No locking.** Two transactions touching the same paths concurrently will
  interfere. Serialise them yourself.
- **`fsync` durability only.** The journal is `fsync`-ed before each change, so
  a process kill or power loss is recoverable, but a lying disk cache or media
  failure is not. Pass `fsync=False` to trade that away for speed.
- **Staged material is visible.** During a transaction, `.filetx-*` entries
  exist alongside your files. A crash leaves them until `filetx undo` runs;
  `filetx.STAGE_PREFIX` is exported so scanners can skip them.
- **Cross-filesystem moves are O(size).** They fall back to copying.

## 7. What I'd change at 100× scale

The design assumes a batch you can hold in memory and a journal you read whole.
At 100× — hundreds of thousands of operations per transaction — three things
break, in this order:

1. **The plan is written to the journal as one record.** A single JSON line
   holding 500k operations has to be serialised and `fsync`-ed before the first
   change happens, and re-parsed in full during recovery. I would stream the
   plan as one record per operation and make `read_journal` incremental.
2. **`fsync` per operation dominates.** At ~1 ms per flush, 500k operations is
   over eight minutes of waiting on the disk. I would batch the flush — group
   commits of N records — accepting a bounded window where the journal lags
   reality, and handle it by making recovery re-verify the tail against the
   filesystem, which the state-checking reverts already support.
3. **Rollback is serial.** Undoing 500k renames one at a time wastes an SSD's
   parallelism. Reverts of operations on disjoint paths could run in a thread
   pool; the ordering constraint is only between operations whose paths nest,
   which is a partial order the planner could compute up front.

What I would *not* change is the staging strategy — it is the part that gets
better with scale, not worse.

---

## References

The write-ahead ordering and the do/undo/purge split follow standard database
recovery practice; ARIES (Mohan et al., 1992) is the canonical description, and
the naming of the three phases here is deliberately borrowed from it. No
implementation was consulted — the filesystem constraints are different enough
that the resemblance is conceptual.

## Licence

MIT — see [LICENSE](LICENSE).
