Metadata-Version: 2.4
Name: fylex
Version: 1.1.3
Summary: Fylex: A production-ready, high-performance file utility tailored for Linux — with smart copy/move, hashing, filters, undo/redo, and conflict resolution.
Author-email: Sivaprasad Murali <sivaprasad.off@gmail.com>
License: MIT
Project-URL: Homepage, https://github.com/Crystallinecore/fylex
Project-URL: Documentation, https://github.com/Crystallinecore/fylex#readme
Project-URL: Source, https://github.com/Crystallinecore/fylex
Project-URL: Tracker, https://github.com/Crystallinecore/fylex/issues
Project-URL: Releases, https://github.com/Crystallinecore/fylex/releases
Keywords: file,copy,move,cli,linux,utility,xxhash,blake3,filter,conflict,backup,undo,redo,hashing,checksum,deduplication,data-integrity,sysadmin,file-management,sync,devtool,cli-tool,python-tool,smart-copy,smart-move,fylex
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Intended Audience :: End Users/Desktop
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: System :: Filesystems
Classifier: Topic :: Utilities
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE.txt
Requires-Dist: xxhash>=3.0.0
Requires-Dist: blake3>=0.4.0
Dynamic: license-file

# Fylex: The Linux File Ninja

[![PyPI](https://img.shields.io/pypi/v/fylex.svg)](https://pypi.org/project/fylex/)
[![Python 3.x](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org/)
[![PyPI Downloads](https://static.pepy.tech/personalized-badge/fylex?period=total\&units=INTERNATIONAL_SYSTEM\&left_color=black\&right_color=green\&left_text=downloads)](https://pepy.tech/projects/fylex)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

[![20250918-2144-Tech-Neon-Design-simple-compose-01k5esk4tqf54rf0vh67m2ph4h.png](https://i.postimg.cc/L6F6V78T/20250918-2144-Tech-Neon-Design-simple-compose-01k5esk4tqf54rf0vh67m2ph4h.png)](https://postimg.cc/WtSvN5rF)

**Fylex** is a **production-ready, Linux-tailored file management tool** that combines the best of features of `rsync`, `cp`, and Python’s `shutil` — but goes **beyond** with:

*  **Smart Copy & Move** with hashing (xxhash, blake3, SHA, MD5)
*  **Advanced conflict resolution** (rename, skip, replace, larger/smaller, newer/older, prompt)
*  **Filters**: regex, glob, exact filename matches, inclusion/exclusion
*  **Safety nets**: undo, redo, backup of deprecated files
*  **Data integrity**: hash verification, SQLite-backed hash cache for deduplication
*  **Metadata preservation**: permissions, timestamps, xattrs, ACLs (Linux-specific)
*  **CLI & Python API** for flexible usage


##
### Feature comparison  

| Feature / Tool                  | Fylex | cp (coreutils) | rsync | shutil (Python stdlib) |
|---------------------------------|-------|----------------|-------|-------------------------|
| Primary purpose                 | Smart local copy/move with safety nets | Basic copy | Fast sync (local/remote) | Library-level file ops |
| Undo / Redo                     | Yes — built-in JSON journaling | No | No | No |
| Hash verification               | Yes — xxhash, blake3, sha256, etc. | No | Partial — checksums optional | No |
| Hash cache (SQLite)             | Yes — avoids rehashing unchanged files | No | No | No |
| Duplicate detection (dest)      | Yes — size + hash | No | Partial — based on size/checksums | No |
| Conflict resolution             | Extensive — rename, replace, skip, newer/older, larger/smaller, prompt | None — overwrite only | Limited — flags like `--backup`, `--suffix` | None |
| Metadata preservation           | Yes — mtime, perms, xattrs, ACLs on Linux | Partial — `-a` preserves many | Partial — `-a` preserves many | Partial — `copystat` only |
| Atomic writes                   | Yes — via `fylex.tmp` | No | Partial — temp options exist | No |
| Logging / audit trail           | Yes — JSON logs per process | No | Partial — verbose logs only | No |
| CLI + Python API                | Yes — both | CLI only | CLI only (bindings exist) | Python API only |
| Delta transfer (network)        | No — local only | No | Yes | No |
| Remote / cloud support          | No — local-first | No | Yes — ssh/rsyncd | No |
| Cross-platform                  | Partial — Linux-first (xattrs/ACL best) | Yes | Yes | Yes |
| Performance (local)             | Very good — uses `copy_file_range` / `sendfile` | Good | Very good — efficient I/O | Moderate |
| Learning curve                  | Moderate — many options | Very low | Moderate to high — many options | Low |
| Best fit                        | Local integrity-critical workflows, reversible ops | Quick one-off copies | Local/remote sync and bandwidth-efficient backups | Small Python scripts |

---

## Strengths

* **Undo / Redo** — Most competitors don’t offer a built-in reversible operation for arbitrary copy/move workflows. This is a major safety feature for power users.
* **JSON audit trail** — Every process writes machine-readable logs that enable reproducibility and automation (and fuel the undo/redo).
* **Hash verification + hash cache** — Optional verification plus an on-disk SQLite cache avoids repeated hashing of unchanged files and speeds repeated operations.
* **Flexible conflict resolution** — Many realistic conflict policies (rename with suffix, replace, choose larger/newer, prompt) are available out-of-the-box.
* **Linux metadata handling** — Attempts to preserve xattrs/ACLs when system tools are available—valuable for server or system admin workflows.
* **Atomic writes & backups** — Writes to a `.tmp` area and moves replaced files to `fylex.deprecated/PROCESS_ID` to avoid data loss during partial operations.
* **Good local performance** — Uses `copy_file_range` or `sendfile` and falls back sensibly for portability.

## Limitations

* **Not a network delta-sync tool** — If you need efficient remote sync over low bandwidth, `rsync` or `rclone` is better (rsync implements delta transfer).
* **Not a full backup system** — For encrypted, deduplicated backups with retention/versioning, choose `borg`/`restic`.
* **Linux-first** — Windows/macOS will work for basic operations, but xattr/ACL preservation and some system calls are Linux-specific; Fylex is intentionally optimized for Linux environments.
* **No built-in remote cloud backends** — If you need native S3/GoogleDrive/OneDrive support, `rclone` is the tool of choice.
* **Single-process / scale** — Current design favors correctness and simplicity; large-scale parallel distributed copying may require additional engineering (e.g., a distributed worker model).
* **Delta-copy library replacement** — Fylex does **not** implement network delta transfers (it focuses on local efficiency and correctness).

##


##  Safety Nets: Reliability First

Fylex is built with **data safety as priority**:

*  **Undo**: Rollback the last copy/move operation (removes created files, restores moved ones).
*  **Redo**: Replay an operation exactly as before.
*  **Backups**: Replaced/conflicting files are moved into `fylex.deprecated/` (per process).
*  **JSON logs**: Every operation is journaled (`json/{process_id}.json` + `.jsonl`).
*  **Verification**: Optional hash verification ensures copy integrity.
*  **Retries**: Up to 5 retries (`MAX_RETRIES`) on hash mismatch.
*  **Protections**: Prevents overwriting itself, copying into subdirectories, or backup loops.

##

## Installation

```bash
pip install fylex
```

Requires Python **3.8+**.
Linux recommended (for full xattr/ACL support).

##

##  Command Line Usage

### Copy files

```bash
fylex copy ~/Downloads ~/Backup --resolve rename --algo xxhash --verify --verbose
```

* Smartly copies, resolves conflicts by renaming, verifies integrity via hash.
* Deprecated/replaced files stored under `~/Backup/fylex.deprecated/PROCESS_ID/`.

### Move files

```bash
fylex move ./data ./archive --resolve newer --match-glob "*.csv"
```

* Moves only `.csv` files, replacing only if source is newer.

### Undo

```bash
fylex undo 1002
```

* Undoes process with ID `1002`.

### Redo

```bash
fylex redo 1002
```

* Replays process with ID `1002`.

##

##  Python API Usage

```python
from fylex import filecopy, filemove, undo, redo

# Copy with filters & conflict resolution
filecopy(
    src="~/Downloads",
    dest="~/Backup",
    resolve="rename",
    algo="xxhash",
    verify=True,
    match_glob="*.jpg",
    verbose=True
)

# Move and preserve metadata
filemove("project/", "archive/", resolve="newer", preserve_meta=True)

# Undo / Redo
undo("1002")
redo("1002")
```
##

##  Function Reference

### `filecopy(src, dest, ...)`

**Description:** Smartly copies files from `src` to `dest` with conflict handling, filters, and safety nets.

| Param             | Type         | Default              | Description                                                                                     |
| ----------------- | ------------ | -------------------- | ----------------------------------------------------------------------------------------------- |
| `src`             | `str\|Path` | required             | Source file or directory                                                                        |
| `dest`            | `str\|Path` | required             | Destination directory                                                                           |
| `resolve`         | `str`        | `"rename"`           | Conflict strategy: `rename`, `replace`, `skip`, `larger`, `smaller`, `newer`, `older`, `prompt` |
| `algo`            | `str`        | `"xxhash"`           | Hash algorithm: `xxhash`, `blake3`, `md5`, `sha256`, `sha512`                                   |
| `chunk_size`      | `int`        | `16 * 1024 * 1024`   | Buffer size (bytes) for reading files                                                           |
| `verbose`         | `bool`       | `True`               | Log operations to stdout                                                                        |
| `dry_run`         | `bool`       | `False`              | Simulate actions without making changes                                                         |
| `summary`         | `str\|Path` | `None`               | Path to copy `fylex.log` summary                                                                |
| `match_regex`     | `str`        | `None`               | Regex pattern to include files                                                                  |
| `match_names`     | `list[str]`  | `None`               | Exact filenames to include                                                                      |
| `match_glob`      | `list[str]`  | `None`               | Glob patterns to include                                                                        |
| `exclude_regex`   | `str`        | `None`               | Regex pattern to exclude files                                                                  |
| `exclude_names`   | `list[str]`  | `None`               | Exact filenames to exclude                                                                      |
| `exclude_glob`    | `list[str]`  | `None`               | Glob patterns to exclude                                                                        |
| `recursive_check` | `bool`       | `False`              | Deduplication check recursively in `dest`                                                       |
| `verify`          | `bool`       | `False`              | Verify file hashes after copying                                                                |
| `has_extension`   | `bool`       | `False`              | Include file extension in deduplication check                                                   |
| `no_create`       | `bool`       | `False`              | Do not create `dest` if it does not exist                                                       |
| `preserve_meta`   | `bool`       | `True`               | Preserve timestamps, permissions, xattrs, ACLs                                                  |
| `backup`          | `str\|Path` | `"fylex.deprecated"` | Folder for deprecated or conflicting files                                                      |
| `recurse`         | `bool`       | `False`              | Traverse subdirectories in `src`                                                                |

**Example:**

```python
filecopy("photos", "photos_backup", resolve="newer", match_glob="*.png", verify=True)
```

##

### `filemove(src, dest, ...)`

Same params as `filecopy`, but moves files instead.
If conflicts exist, originals are moved into deprecated folders within `src` or `dest` depending on the origin of the file being deprecated.

##

### `undo(p_id, verbose=True, force=False)`

Rollback a process by ID.

| Param     | Type   | Description                             |
| --------- | ------ | --------------------------------------- |
| `p_id`    | `str`  | Process ID (JSON log ID)                |
| `verbose` | `bool` | Enable logs                             |
| `force`   | `bool` | Continue undo even if some entries fail |
| `summary`   | `str \| Path` | Path to copy fylex.log summary |

##

### `redo(p_id, verbose=True, force=False)`

Replay a process by ID. Same parameters as `undo`.

##

##  Example Use Cases

* **Daily backup with safety nets**

  ```bash
  fylex copy ~/work ~/backup --resolve newer --verify --summary=backup.log
  ```

* **Selective move**

  ```bash
  fylex move ./data ./archive --match-regex ".*2025.*\.csv"
  ```

* **Quick rollback**

  ```bash
  fylex undo 1021
  ```

* **Replay last operation for reproducibility**

  ```bash
  fylex redo 1021
  ```

##

## Internals

* **Hash cache:** Stored in `file_cache.db` (SQLite). All file hashes computed with the chosen algorithm (`xxhash`, `blake3`, etc.) are cached here to avoid redundant computation.
* **Logging:** JSON logs are stored in the `json/` folder in both `.jsonl` (line-delimited JSON) and `.json` formats. These logs track all file operations, conflicts, renames, and metadata preservation.
* **Temporary files:** During copy or move operations, temporary files are written to `dest/fylex.tmp/` to ensure atomic writes. Only after successful completion are files renamed to their final names.
* **Backups / Deprecated files:** Conflicting or replaced files are moved to `fylex.deprecated/{process_id}/`.

  * If the source file is within `src`, the backup is created at `src/fylex.deprecated/{process_id}/`.
  * If the source file is within `dest`, the backup goes to `dest/fylex.deprecated/{process_id}/`.
* **Move operations:** Fylex implements move by first copying the file to the destination, verifying the integrity of the copy (via hash comparison if `verify=True`), and then deleting the source file. This ensures safety and prevents data loss.
* **Conflict resolution:** Handled according to the `resolve` strategy (`rename`, `replace`, `skip`, `larger`, `smaller`, `newer`, `older`, `prompt`). Conflicting files are either renamed, replaced, skipped, or backed up depending on the configuration.
* **Metadata preservation:** File timestamps, permissions, extended attributes (xattrs), and ACLs are preserved if `preserve_meta=True`.
* **Recursive deduplication:** When `recursive_check=True`, Fylex scans the destination folder to detect duplicates using cached hashes. Optional `has_extension` filtering allows deduplication by considering file extensions.
* **Filters:** Files can be included or excluded using regex patterns (`match_regex`, `exclude_regex`), exact names (`match_names`, `exclude_names`), or glob patterns (`match_glob`, `exclude_glob`).
* **Atomicity and safety:** Temporary files, hash verification, and backup mechanisms make all operations safe against crashes or interruptions.


##

## License

MIT © 2025 Sivaprasad Murali

---

✨ With **Fylex**, file management on Linux is no longer just copying and moving — it’s **safe, verifiable, reversible, and smart**.

---

