Metadata-Version: 2.4
Name: filefilter
Version: 0.2.7
Summary: Filter files in a directory tree based on configurable glob rules (JSON, YAML, or TOML).
Author-email: "Ioannis D (devcoons)" <support@devcoons.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/devcoons/filefilter
Project-URL: Issues, https://github.com/devcoons/filefilter/issues
Keywords: filefilter
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=5.4
Requires-Dist: tomli>=1.2.2; python_version < "3.11"
Provides-Extra: dev
Requires-Dist: pip-tools; extra == "dev"
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Dynamic: license-file

# filefilter - File Filter Library

![PyPI - Version](https://img.shields.io/pypi/v/filefilter?style=for-the-badge)
![PyPI - Python Version](https://img.shields.io/pypi/pyversions/filefilter?style=for-the-badge)
![PyPI - License](https://img.shields.io/pypi/l/filefilter?style=for-the-badge)
![PyPI - Wheel](https://img.shields.io/pypi/wheel/filefilter?style=for-the-badge&color=%23F0F)
![PyPI - Downloads](https://img.shields.io/pypi/dm/filefilter?style=for-the-badge)

A small, cross-platform Python library for selecting files from a directory tree using **JSON, YAML, or TOML include/exclude filters**.  
It focuses on predictable pattern logic rather than the underlying filesystem walk.

---

##  Features 

- **Pattern-driven filtering** — no manual path checks.
- **JSON, YAML, or TOML**, passed as text or as a file path.
- **One filter, or an array of filters.** Each array entry is applied on its own. A file is kept when any entry selects it.
- **Case-insensitive matching** across all platforms.
- **Supports rich glob-style patterns**:
  - `*` → one or more characters  
  - `**` → zero or more characters  
  - Works in both directory and filename segments.
- **Anchored directory semantics**:
  - No prefix → root-anchored
  - `*/` → exactly N levels below root
  - `**/` → anywhere
- **Explicit extension filtering** (`["py", "tar.gz", ".*"]`)
- **Fine-grained include/exclude precedence**
- **Symlinks ignored** (never followed)


### Folder patterns

| Group                             | Example Pattern                 | Meaning (actual behavior)                                                   | Example Matches                                            | Notes / Clever use-cases                            |
| :-------------------------------- | :------------------------------ | :-------------------------------------------------------------------------- | :--------------------------------------------------------- | :-------------------------------------------------- |
| **Root-anchored**                 | `/folder`                       | Match directory named `folder` directly under the root directory            | `/folder`, `/folder/file.txt`                              | Root only                                           |
|                                   | `/folder/*`                     | Match subdirectories **one level** under `/folder`                          | `/folder/a`                                                | Single depth                                        |
|                                   | `/folder/*/*`                   | Match subdirectories **two levels** under `/folder`                         | `/folder/a/b`                                              | Exact depth control                                 |
|                                   | `/folder/**`                    | Match `/folder` and all its subdirectories recursively                      | `/folder/x/y`                                              | Deep traversal                                      |
|                                   | `/folder/**/a`                  | Match any nested `a` under `/folder`                                        | `/folder/a`, `/folder/x/y/a`                               | Anchored root start                                 |
| **Relative (no prefix)**          | `folder`                        | Same as `/folder`, relative to `root_dir`                                   | `/folder`                                                  | Root alias                                          |
|                                   | `folder/*`                      | Match subdirectories one level below `folder`                               | `/folder/sub`                                              | Controlled depth                                    |
|                                   | `folder/*/*`                    | Match directories two levels below `folder`                                 | `/folder/a/b`                                              | Exact two levels                                    |
|                                   | `folder/**`                     | Match `folder` and all subdirectories recursively                           | `/folder/x/y`                                              | Recursive within folder                             |
| **Single-level leading wildcard** | `*/folder`                      | Match any `folder` one level below any top-level dir                        | `/src/folder`, `/data/folder`                              | “Exactly one level up”                              |
|                                   | `*/folder/**`                   | Same as above, but include everything under each matched folder             | `/src/folder/x/y`                                          | Deep from one-level up                              |
| **Multi-level leading wildcard**  | `*/*/folder`                    | `folder` that is **two levels** below any top-level dir                     | `/apps/web/folder`, `/lib/core/folder`                     | “Exactly two levels up”                             |
|                                   | `*/*/*/folder`                  | `folder` that is **three levels** below any top-level dir                   | `/a/b/c/folder`                                            | Scale to N levels: repeat `*/`                      |
|                                   | `*/*/*/*/folder/**`             | Same as above, then **recursive** under `folder`                            | `/a/b/c/d/folder/x/y`                                      | Useful in deep monorepos                            |
| **Recursive leading wildcard**    | `**/folder`                     | Match any directory named `folder` at any depth                             | `/folder`, `/a/b/c/folder`                                 | “Any depth before”                                  |
|                                   | `**/folder/**`                  | Match any directory `folder` and all subdirectories under it                | `/folder/x`, `/x/y/folder/z`                               | Fully recursive                                     |
|                                   | `**`                            | Match **all directories**                                                   | (everything)                                               | Wildcard all                                        |
|                                   | `**/**`                         | Same as `**`                                                                | (everything)                                               | Redundant doublestar                                |
| **Mixed / Middle wildcards**      | `folder/**/sub`                 | Match any `sub` folder nested inside `folder` (any depth gap)               | `/folder/x/y/sub`                                          | “Any depth in between”                              |
|                                   | `*/folder/**/sub`               | Match any `sub` inside any `folder` one level below any root dir            | `/a/folder/sub`, `/a/folder/x/y/sub`                       | Combined leading + middle wildcards                 |
|                                   | `**/folder/another`             | Match any `another` directly inside any `folder` at any depth               | `/folder/another`, `/x/folder/another`                     | Common nested pattern                               |
|                                   | `**/pkg/*`                      | Match **one level after** any `pkg` at any depth                            | `/pkg/a`, `/x/y/pkg/z`                                     | Limit scope to immediate children                   |
|                                   | `**/pkg/*/*`                    | Match **two levels after** any `pkg` at any depth                           | `/pkg/a/b`, `/x/y/pkg/z/t`                                 | Exact depth after target                            |
|                                   | `/modules/*/test`               | A `test` dir exactly one level under `/modules/<name>`                      | `/modules/auth/test`, `/modules/pay/test`                  | Standard per-module test folder                     |
|                                   | `/apps/**/dist`                 | Any `dist` inside `/apps`                                                   | `/apps/web/dist`, `/apps/mobile/a/b/dist`                  | Clean build artifacts                               |
| **Exact middle gaps**            | `*/*/folder/*/another_folder`   | **Exactly 1** segment between `folder` and `another_folder`                 | `a/b/folder/c/another_folder`                              | Middle `*` is one segment                           |
|                                   | `*/*/folder/**/another_folder`  | **Any number (0+)** of segments between them                                | `a/b/folder/another_folder`, `a/b/folder/x/another_folder` | Use when the gap can vary                           |
|                                   | `*/*/folder/*/*/another_folder` | **Exactly 2** segments between them                                         | `a/b/folder/x/y/another_folder`                            | Chain `*` for exact middle depth                    |
|                                   | `**/services/**/migrations/*`   | Immediate children under any `migrations` inside any `services` subtree     | `/x/services/a/migrations/001`, `/services/m/2`            | “Find migration versions but not nested subfolders” |
|                                   | `**/features/*/*`               | Exactly **two** levels under any `features` directory                       | `/a/features/x/y`, `/features/p/q`                         | Depth-limited collection                            |
|                                   | `/packages/*/**/dist`           | Any `dist` under any direct child of `/packages`                            | `/packages/core/dist`, `/packages/ui/x/y/dist`             | Works across package families                       |
| **Literal / Edge cases**          | `**/folder/***/anotherfolder`   | Requires a **literal** `***` directory between `folder` and `anotherfolder` | `/folder/***/anotherfolder`                                | `***` is literal (not wildcard)                     |
|                                   | `/folder/***/anotherfolder`     | Same, but anchored at root                                                  | `/folder/***/anotherfolder`                                | Rare in practice                                    |
| **Edge / invalid**                | `/` (or empty)                  | Ignored / invalid after normalization                                       | —                                                          | Stripped by normalization                           |

### File patterns


| Group                             | Example Pattern             | Meaning (actual behavior)                                                                | Example Matches                                                | Notes / Clever use-cases                                                  |
| :-------------------------------- | :-------------------------- | :--------------------------------------------------------------------------------------- | :------------------------------------------------------------- | :------------------------------------------------------------------------ |
| **Root-anchored**                 | `/file.py`                  | Match `file.py` **at project root**                                                      | `/file.py`                                                     | Exact root match                                                          |
|                                   | `/README.*`                 | Files starting with `README.` at root, with **at least one** char after the dot          | `/README.md`, `/README.rst`                                    | Because filename `*` = **1+** chars                                       |
|                                   | `/README.**`                | Files starting with `README.` at root, allowing **zero or more** chars after the dot     | `/README`, `/README.md`, `/README.rst`                         | Filename `**` can match empty                                             |
|                                   | `/.*`                       | Dotfiles at root with **at least one** char after the dot                                | `/.env`, `/.gitignore`                                         | Matches hidden files                                                      |
|                                   | `/folder/*.py`              | Any `.py` directly under `/folder`                                                       | `/folder/a.py`, `/folder/test.py`                              | One level below `/folder`                                                 |
|                                   | `/folder/**/test_*.py`      | `test_*.py` anywhere under `/folder`                                                     | `/folder/test_a.py`, `/folder/x/y/test_utils.py`               | Classic recursive test selector                                           |
| **Relative (no prefix)**          | `Makefile`                  | Match `Makefile` at project root                                                         | `/Makefile`                                                    | Case-insensitive                                                          |
|                                   | `*.toml`                    | Any `.toml` file directly under the root directory                                       | `/pyproject.toml`                                              | Only at root; use `**/*.toml` for recursive                               |
|                                   | `**/*.toml`                 | Any `.toml` file **anywhere**                                                            | `/pyproject.toml`, `/pkg/a/b/config.toml`                      | Recursive                                                                 |
|                                   | `folder/**/*.md`            | Any `.md` anywhere under `folder`                                                        | `/folder/README.md`, `/folder/docs/a/b/guide.md`               | Folder-scoped recursion                                                   |
| **Single-level leading wildcard** | `*/Dockerfile`              | `Dockerfile` exactly **one** directory deep                                              | `/api/Dockerfile`, `/web/Dockerfile`                           | “One level up”                                                            |
|                                   | `*/LICENSE*`                | Files whose name starts with `LICENSE` exactly one level deep                            | `/pkg/LICENSE`, `/lib/LICENSE-MIT`                             | Filename `*` = 1+ chars (so `LICENSE` alone won’t match; use `LICENSE**`) |
| **Multi-level leading wildcard**  | `*/*/Makefile`              | `Makefile` exactly **two** directories deep                                              | `/x/y/Makefile`, `/apps/web/Makefile`                          | Repeat `*/` to require N levels                                           |
|                                   | `*/*/*/package.json`        | `package.json` exactly **three** directories deep                                        | `/a/b/c/package.json`                                          | Deep monorepo layouts                                                     |
| **Recursive leading wildcard**    | `**/Makefile`               | `Makefile` at **any depth**                                                              | `/Makefile`, `/a/Makefile`, `/a/b/c/Makefile`                  | Ignore structure depth                                                    |
|                                   | `**/*.py`                   | Any `.py` file at any depth                                                              | `/a.py`, `/pkg/mod/x.py`, `/x/y/z/__init__.py`                 | Global Python selection                                                   |
|                                   | `**`                        | **All files** (filename `**` matches empty, so any name)                                 | Every file                                                     | Combine with directory excludes to prune                                  |
| **Mixed / middle wildcards**      | `src/**/test_*.py`          | Any `test_*.py` under `src`, at any depth                                                | `/src/test_a.py`, `/src/unit/core/test_utils.py`               | Common test pattern                                                       |
|                                   | `**/migrations/*`           | Any file **one level under** a `migrations` dir at any depth                             | `/app/migrations/001.sql`, `/x/y/migrations/init.py`           | Excludes deeper levels; add `/*/*` for exactly two                        |
|                                   | `**/migrations/*/*`         | Any file **two levels under** a `migrations` dir                                         | `/app/migrations/v1/001.sql`                                   | Exact depth after target                                                  |
|                                   | `**/assets/**/*.png`        | Any `.png` somewhere under any `assets` dir                                              | `/assets/logo.png`, `/x/assets/img/icons/a.png`                | Nested assets                                                             |
|                                   | `/packages/*/**/dist/*.js`  | `.js` directly in any `dist` below any direct child of `/packages`                       | `/packages/core/dist/index.js`, `/packages/ui/x/y/dist/app.js` | Package builds                                                            |
|                                   | `**/pkg/*/*/index.*`        | `index.<ext>` exactly two levels under any `pkg` dir                                     | `/pkg/a/b/index.js`, `/x/y/pkg/z/t/index.html`                 | Depth-limited bundle entry                                                |
| **Clever filename globs**         | `**/file*.py`               | Filenames starting with `file` then **≥1** char, ending `.py`                            | `/fileA.py`, `/x/y/file_utils.py`                              | Does **not** match `/file.py` (use `file**.py` for optional tail)         |
|                                   | `**/file**.py`              | `file` plus **0+** chars, `.py`                                                          | `/file.py`, `/x/fileA.py`, `/x/y/file_utils.py`                | Optional tail                                                             |
|                                   | `**/*test*.py`              | Any file with `test` substring and `.py`                                                 | `/test_main.py`, `/x/y/unittest_tools.py`                      | Broad test sweep                                                          |
|                                   | `**/.env**`                 | `.env` or `.env.<suffix>` anywhere                                                       | `/.env`, `/app/.env.local`, `/x/.envrc`                        | Dotfile families                                                          |
|                                   | `**/*.tar.**`               | `.tar` followed by any (even empty) extension                                            | `/a.tar.gz`, `/b/c.tar.bz2`, `/d/e.tar.`                       | `**` in filename can match empty                                          |
|                                   | `**/*.*`                    | Files with a dot and **at least one** char extension                                     | `/a.txt`, `/b/c.tar.gz`                                        | “Has extension” filter                                                    |
|                                   | `**/*.`                     | Filenames ending with a dot                                                              | `/strange.`                                                    | Rare but supported                                                        |
| **Directory gaps**                | `*/*/folder/*/another.py`   | **Exactly 1** directory between `folder` and `another.py`, and 2 before `folder`         | `/a/b/folder/c/another.py`                                     | Middle `*` is one segment                                                |
|                                   | `*/*/folder/**/another.py`  | **Any number (0+)** directories between `folder` and `another.py`, and 2 before `folder` | `/a/b/folder/another.py`, `/a/b/folder/x/y/another.py`         | Flexible gap                                                              |
|                                   | `*/*/folder/*/*/another.py` | **Exactly 2** directories between `folder` and `another.py`, and 2 before `folder`       | `/a/b/folder/x/y/another.py`                                   | Chain `*` for exact middle depth                                          |
| **Exact names at any depth**      | `**/LICENSE**`              | `LICENSE` with optional suffix at any depth                                              | `/LICENSE`, `/pkg/LICENSE-MIT`                                 | Use `**` to allow empty/extra chars                                       |
|                                   | `**/Dockerfile`             | `Dockerfile` at any depth                                                                | `/Dockerfile`, `/services/api/Dockerfile`                      | Service Dockerfiles                                                       |
|                                   | `**/Makefile`               | `Makefile` at any depth                                                                  | `/Makefile`, `/lib/x/Makefile`                                 | Build roots                                                               |
| **Edge / literal cases**          | `**/folder/***/file.txt`    | Requires a **literal** directory named `***` between `folder` and file                   | `/folder/***/file.txt`                                         | `***` is literal, not a wildcard                                          |
|                                   | `folder/*/file.txt`         | **Exactly 1** directory between `folder` and file                                        | `/folder/a/file.txt`                                           | Middle `*` is one directory                                                |
|                                   | `folder/**/file.txt`        | Any depth between `folder` and file                                                      | `/folder/file.txt`, `/folder/a/b/file.txt`                     | Versatile                                                                 |
| **Gotchas**                       | `file*.py` (at root)        | Requires at least one char after `file`                                                  | Matches `/fileA.py`; **does not** match `/file.py`             | Use `file**.py` if `file.py` should match                                 |
|                                   | `LICENSE*`                  | Requires at least one char after `LICENSE`                                               | Matches `LICENSE-MIT`; **not** `LICENSE`                       | Use `LICENSE**` to include bare `LICENSE`                                 |
|                                   | `?`                         | Not supported                                                                            | —                                                              | Stick to `*` and `**`                                                     |






##  Decision Order (Inclusion / Exclusion Logic)

When scanning files, the library applies each filter object in this exact sequence. If `filters` is an array, that sequence runs once per entry, and a file is kept when any entry selects it.

1️⃣ **Extension excludes (hard)**  
   - If the file extension matches `exclude.extensions` → **excluded immediately**.  
   - Not overridden by `odirs`, `ofiles`, or any other include rule.

2️⃣ **Pass 1 — Scope (`include.dirs`, `include.files`)**  
   - If any `include.dirs` or `include.files` patterns are set, the file must match **at least one** of them.  
   - Otherwise → **excluded** (excludes are not evaluated).  
   - `include.odirs` and `include.ofiles` do **not** define scope.

3️⃣ **Pass 2 — Path excludes (`exclude.dirs`, `exclude.files`)**  
   - If the file path matches `exclude.files`, or its parent directory matches `exclude.dirs` → **marked excluded**.

4️⃣ **Pass 3 — Overrides (`include.odirs`, `include.ofiles`)**  
   - If pass 2 excluded the file, a matching `include.odirs` or `include.ofiles` pattern → **exclude is undone**.  
   - If no override matches → **excluded**.  
   - Any matching override wins (no specificity scoring).

   > Use `include.dirs` / `include.files` for normal scoping.  
   > Use `include.odirs` / `include.ofiles` only to carve exceptions out of pass-2 excludes.

5️⃣ **Include-file fast path**  
   - If the file matches any `include.files` pattern → **included immediately**,  
     even if its extension isn’t in `include.extensions`.

6️⃣ **Extension whitelist**  
   - If `include.extensions` is not empty, only matching extensions are included.

7️⃣ **Everything else**  
   - Included by default.

### Dry-run / rule hit report

`dry_run(rules)` walks the tree like `scan`, but also records how many files each rule pattern matched. Rule keys use the form `category:pattern`, e.g. `include.dirs:**`, `exclude.files:**/test_*.py`.

```python
from filefilter import dry_run, load

rules = load(config, base="cwd")
report = dry_run(rules)

report.scanned          # files walked
report.included         # paths that would be selected
report.excluded         # scanned - included
report.hits             # {"include.dirs:**": 42, ...}
report.was_hit("exclude.dirs:**/build/**")
report.has_rule("include.extensions:.py")   # configured (may be 0)
report.count("include.extensions:.py")
```

Hits count **pattern matches per file** during the walk. Path rules (`dirs`, `files`, …) match on every scanned file. `include.extensions` only counts files that **reach the extension whitelist** (not skipped by `include.files` fast path or earlier excludes). `exclude.extensions` matches on any scanned file. Configured extension patterns always appear in `report.hits` (with count `0` when unused). Keys use parsed form, e.g. `include.extensions:.py` for a configured `py` extension. When `filters` is an array of more than one filter, each filter is counted on its own and the key is prefixed with its index, e.g. `filters[0].include.extensions:.py`. A single object, and a one-element array, keep the unprefixed keys.

---

##  Configuration Schema

Each filter object has this shape. `filters` itself is either **one object** or an **array of objects**.

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": [],
      "odirs": [],
      "ofiles": [],
      "files": [],
      "extensions": []
    },
    "exclude": {
      "dirs": [],
      "files": [],
      "extensions": []
    }
  }
}
```

| Field | Description |
|-------|--------------|
| `root_dir` | Base directory (absolute or relative). |
| `filters.include.dirs` | Directory scope patterns. When set, files must live under a matching directory. Never overrides excludes. |
| `filters.include.files` | File scope patterns. When set (alone or with dirs), matching files pass pass 1. Also forces inclusion (skips extension whitelist) once later passes succeed. Does not override excludes. |
| `filters.include.odirs` | Optional override directory patterns (defaults to `[]`). Undo pass-2 `exclude.dirs` / `exclude.files` when matched. Do not define scope. |
| `filters.include.ofiles` | Optional override file patterns (defaults to `[]`). Undo pass-2 excludes when matched. Do not define scope or skip extension whitelist. |
| `filters.include.extensions` | Extension whitelist. |
| `filters.exclude.*` | Same structure, but acts as exclusion filters. |

On an array, the same fields sit on each entry (`filters[0].include.dirs`, and so on).

### Single filter

`filters` is one object. In JSON, YAML, and TOML that is the same structure. A one-element array selects the same files. The two JSON documents below are equivalent:

```json
{
  "root_dir": ".",
  "filters": {
    "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
    "exclude": { "dirs": ["__pycache__"], "files": [], "extensions": [] }
  }
}
```

```json
{
  "root_dir": ".",
  "filters": [{
    "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
    "exclude": { "dirs": ["__pycache__"], "files": [], "extensions": [] }
  }]
}
```

### Multiple filters

`filters` is an array of objects, in JSON, YAML, or TOML. The array is not merged into one ruleset. Each object is applied on its own with the decision order above, then the selected paths are combined: a file is kept when **any** filter selects it. An exclude in one filter does not remove a file that another filter selected.

```json
{
  "root_dir": ".",
  "filters": [
    {
      "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
      "exclude": { "dirs": ["**/skip/**"], "files": [], "extensions": [] }
    },
    {
      "include": { "dirs": [], "files": ["**/skip/keep.py"], "extensions": [] },
      "exclude": { "dirs": [], "files": [], "extensions": [] }
    }
  ]
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `src/app.py` | ✅ | first filter selects `.py` files |
| `docs/guide.md` | ❌ | neither filter selects it |
| `skip/other.py` | ❌ | first filter excludes `**/skip/**`; second filter does not name this file |
| `skip/keep.py` | ✅ | second filter selects it, even though the first filter excludes `skip` |

With more than one filter, dry-run keys are prefixed with the filter index: `filters[0].include.dirs:**`, `filters[1].include.files:**/skip/keep.py`. A single filter keeps the unprefixed keys.

```yaml
root_dir: "."
filters:
  - include:
      dirs: ["**"]
      extensions: ["py"]
    exclude:
      dirs: ["**/skip/**"]
  - include:
      files: ["**/skip/keep.py"]
    exclude: {}
```

```toml
root_dir = "."

[[filters]]
[filters.include]
dirs = ["**"]
extensions = ["py"]
[filters.exclude]
dirs = ["**/skip/**"]

[[filters]]
[filters.include]
files = ["**/skip/keep.py"]
[filters.exclude]
```

---

##  Pattern Rules Recap

| Symbol | Meaning |
|:-------|:--------|
| `*` | one or more characters (no slashes) |
| `**` | zero or more characters (can cross dirs) |
| `*ABC` / `ABC*` | within a **directory name**, `*` = 1+ chars (e.g. `Myhello` matches `*hello`, not bare `hello`) |
| `**ABC` / `ABC**` | within a **directory name**, `**` = 0+ chars (e.g. `hello` and `Myhello` both match `**hello`) |
| Leading `**/` | match anywhere in the tree |
| Leading `*/` | match exactly N levels below root |
| Trailing `/**` | match directory and all descendants |
| `***` (3+ asterisks) | literal directory name, not a wildcard |
| `.ext` or `ext` | file extension match (case-insensitive) |
| `.*` in extensions | any non-empty extension |

---

##  Examples (Practical Scenarios)

###  Example 1 — Include all `.py` files anywhere

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["**"],
      "files": [],
      "extensions": ["py"]
    },
    "exclude": {
      "dirs": [],
      "files": [],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `main.py` | ✅ | `.py` extension matches |
| `src/app.py` | ✅ | `.py` extension matches |
| `docs/readme.md` | ❌ | wrong extension |
| `a/b/c/module.py` | ✅ | `.py` file anywhere |

> ✅ All `.py` files anywhere in the project.

---

###  Example 2 — Only Python files in root

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": [],
      "files": ["*.py"],
      "extensions": []
    },
    "exclude": {
      "dirs": [],
      "files": [],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `main.py` | ✅ | matches `*.py` at root |
| `helper.py` | ✅ | matches `*.py` at root |
| `src/app.py` | ❌ | subdirectory, not root |
| `src/helper.py` | ❌ | subdirectory, not root |

> ✅ Matches `.py` files **only in the project root**, not deeper folders.

---

###  Example 3 — Include only certain substructure

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": [],
      "files": ["*/hi/**/hello.py"],
      "extensions": []
    },
    "exclude": {
      "dirs": [],
      "files": [],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `a/hi/hello.py` | ✅ | matches `*/hi/**/hello.py` |
| `a/hi/x/hello.py` | ✅ | `hi` one level below root, deeper path allowed |
| `hi/hello.py` | ❌ | missing leading dir before `hi` |
| `a/b/hi/hello.py` | ❌ | `hi` too deep |
| `a/hi/hello.txt` | ❌ | wrong filename |

> ✅ Collects any `hello.py` file under a folder named `hi` that is **exactly one level below root**, with any depth below that.

---

###  Example 4 — Exclude specific pattern while including `.py` files

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["**"],
      "files": [],
      "extensions": [".py"]
    },
    "exclude": {
      "dirs": [],
      "files": ["*/hi/**/hello.py"],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `main.py` | ✅ | `.py`, no exclusion hit |
| `src/lib/module.py` | ✅ | `.py`, no exclusion hit |
| `a/hi/hello.py` | ❌ | excluded by `*/hi/**/hello.py` |
| `src/hi/utils/hello.py` | ❌ | matches exclusion pattern |
| `hi/hello.py` | ✅ | not excluded (no leading dir before `hi`) |

> ✅ Includes all `.py` files except those named `hello.py` inside a `hi` folder that is **one directory below root**.

---

###  Example 5 — Directory pattern focus

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["**/myfolder/**"],
      "files": [],
      "extensions": ["py"]
    },
    "exclude": {
      "dirs": ["**/__pycache__/**"],
      "files": [],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `src/myfolder/x/hello.py` | ✅ | inside `myfolder` |
| `src/myfolder/sub/a.py` | ✅ | inside `myfolder` |
| `src/myfolder/__pycache__/cached.py` | ❌ | excluded dir |
| `src/other/hello.py` | ❌ | outside `myfolder` |

> ✅ Collects `.py` files anywhere under directories named `myfolder`, excluding `__pycache__`.

---

###  Example 6 — Several rules in one filter

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["src/**", "scripts"],
      "files": ["**/*.sh", "*.py"],
      "extensions": ["py", "sh"]
    },
    "exclude": {
      "dirs": ["**/__pycache__/**", "build"],
      "files": ["*/legacy/**"],
      "extensions": ["log"]
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `src/main.py` | ✅ | `.py`, included dir |
| `scripts/setup.sh` | ✅ | matched by `**/*.sh` |
| `build/config.py` | ❌ | excluded dir `build` |
| `src/legacy/module.py` | ❌ | excluded by `*/legacy/**` |
| `docs/readme.md` | ❌ | wrong extension |
| `src/utils/tool.log` | ❌ | excluded by extension |

> ✅ Keeps `.py` or `.sh` files under `src` or `scripts`, ignoring logs, builds, and legacy directories.

---

###  Example 7 — Exclude a subtree, except an explicit include path

Exclude everything under `KLM`, but still collect files under `KLM/ABC`.  
Use `include.dirs` for the broad scope and `include.odirs` for the exception:

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["**"],
      "odirs": ["**/KLM/ABC/**"],
      "files": [],
      "extensions": ["c"]
    },
    "exclude": {
      "dirs": ["**/KLM/**"],
      "files": [],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `src/foo.c` | ✅ | broad `**` include + `.c` extension |
| `src/KLM/skip.c` | ❌ | `**/KLM/**` exclude; no matching `odirs` |
| `src/KLM/ABC/keep.c` | ✅ | `**/KLM/ABC/**` odir undoes `**/KLM/**` exclude |
| `src/x/KLM/ABC/deep/keep.c` | ✅ | odir exception works at any depth |
| `src/KLM/ABC/keep.txt` | ❌ | wrong extension (still inside the allowed dir) |

The same technique applies to other “exclude a path part, except one branch” cases — for example, exclude `**/ABC/**` but keep `**/ABC/KLM/**` via `include.odirs: ["**/ABC/KLM/**"]` alongside `include.dirs: ["src/**"]`.

---

###  Example 8 — Exclude file patterns, except specific files

Exclude all `test_*.py` files, but keep `test_keep.py`:

```json
{
  "root_dir": ".",
  "filters": {
    "include": {
      "dirs": ["**"],
      "ofiles": ["**/test_keep.py"],
      "files": [],
      "extensions": ["py"]
    },
    "exclude": {
      "dirs": [],
      "files": ["**/test_*.py"],
      "extensions": []
    }
  }
}
```

| File Path | Result | Reason |
|------------|---------|--------|
| `src/app.py` | ✅ | broad include + `.py` extension |
| `src/test_foo.py` | ❌ | `**/test_*.py` exclude; no matching `ofiles` |
| `src/test_keep.py` | ✅ | `**/test_keep.py` ofile undoes `**/test_*.py` exclude |
| `nested/test_keep.py` | ✅ | ofile exception works at any depth |

---

## Summary of Behavior

- `exclude.extensions` are always applied first (hard exclude).  
- **Pass 1:** `include.dirs` / `include.files` define scope (when set).  
- **Pass 2:** `exclude.dirs` / `exclude.files` remove files.  
- **Pass 3:** `include.odirs` / `include.ofiles` undo pass-2 excludes when matched.  
- `include.dirs` and `include.files` **never** override excludes — use `odirs` / `ofiles`.  
- A matching `include.files` pattern forces inclusion (skips extension whitelist) after passes 1–3.  
- If any include filters exist, at least one must match (inclusion gating).  
- When `filters` is an array, that decision order runs **once per entry**. The results are combined: a file is selected if any entry selects it.  
- Extensions act as a **final whitelist** when not bypassed by `include.files`.  
- Matching is **case-insensitive** and normalized.  
- `**` can span directory boundaries, not just characters.

---

##  Configuration formats

`load` and `select` take the same configuration as **JSON, YAML, or TOML**. Pass the document text, or a path to a `.json`, `.yaml`, `.yml`, or `.toml` file.

- A file is parsed by its extension.
- Document text is recognized automatically. Valid JSON is parsed as JSON.
- A path is relative to the current working directory. `base` only resolves a relative `root_dir`.
- In YAML, plain values stay strings, so patterns such as `01`, `on`, `yes`, and `null` are not rewritten.
- PyYAML is installed with the package. TOML uses the standard library on Python 3.11 and later, and `tomli` on older Python.

A single filter is one object in every format. These three documents select the same files:

```json
{
  "root_dir": ".",
  "filters": {
    "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
    "exclude": { "dirs": ["__pycache__"], "files": [], "extensions": [] }
  }
}
```

```yaml
root_dir: "."
filters:
  include:
    dirs: ["**"]
    files: []
    extensions: ["py"]
  exclude:
    dirs: ["__pycache__"]
    files: []
    extensions: []
```

```toml
root_dir = "."

[filters.include]
dirs = ["**"]
files = []
extensions = ["py"]

[filters.exclude]
dirs = ["__pycache__"]
files = []
extensions = []
```

An array of filters is the same in every format. JSON uses a normal array. YAML uses a list under `filters:`. TOML uses `[[filters]]`. Those documents are in [Multiple filters](#multiple-filters).

```python
# Text, or a path. The extension selects the parser for a path.
matched_files = select("filters.json")
matched_files = select("filters.yaml")
matched_files = select("filters.toml")
rules = load("filters.json", base="cwd")
```

---

##  Example Usage

```python
from filefilter import *

# Single filter: filters is one JSON object.
cfg_single = """{
    "root_dir": ".",
    "filters": {
        "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
        "exclude": { "dirs": ["__pycache__"], "files": [], "extensions": [] }
    }
}"""

# Same single filter, written as a one-element array.
cfg_single_list = """{
    "root_dir": ".",
    "filters": [{
        "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
        "exclude": { "dirs": ["__pycache__"], "files": [], "extensions": [] }
    }]
}"""

# Multiple filters: each object runs on its own, then the matches are combined.
cfg_multi = """{
    "root_dir": ".",
    "filters": [
        {
            "include": { "dirs": ["**"], "files": [], "extensions": ["py"] },
            "exclude": { "dirs": ["**/skip/**"], "files": [], "extensions": [] }
        },
        {
            "include": { "dirs": [], "files": ["**/skip/keep.py"], "extensions": [] },
            "exclude": { "dirs": [], "files": [], "extensions": [] }
        }
    ]
}"""

# select(cfg_single) and select(cfg_single_list) return the same paths.
matched_files = select(cfg_single)
matched_files = select(cfg_multi)

rules = load(cfg_single, base="cwd")
matched_files = scan(rules)

# Dry-run: same selection plus per-rule hit counts.
# A single filter uses keys like include.dirs:**.
# Multiple filters prefix the index: filters[0].include.dirs:**.
report = dry_run(rules)
print(report.scanned, report.excluded, report.count("include.dirs:**"))

for f in matched_files:
    print(f)
```

---

##  License

MIT License © 2025 Ioannis D. (devcoons)
