Metadata-Version: 2.4
Name: abstract_search
Version: 0.1.1
Summary: Standalone string search across files and directories with allow/exclude filtering; stdlib-only core that reads code, text, pdf, office, and archive-based formats.
Author-email: putkoff <partners@abstractendeavors.com>
License: MIT
Project-URL: Homepage, https://github.com/AbstractEndeavors/abstract_search
Keywords: search,grep,files,filters,text-extraction,dependency-free
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Provides-Extra: pdf
Requires-Dist: pdfplumber; extra == "pdf"
Provides-Extra: ocr
Requires-Dist: pytesseract; extra == "ocr"
Requires-Dist: pdf2image; extra == "ocr"
Provides-Extra: xls
Requires-Dist: xlrd; extra == "xls"
Provides-Extra: xlsb
Requires-Dist: pyxlsb; extra == "xlsb"
Provides-Extra: dbf
Requires-Dist: dbfread; extra == "dbf"
Provides-Extra: parquet
Requires-Dist: pyarrow; extra == "parquet"
Provides-Extra: all
Requires-Dist: pdfplumber; extra == "all"
Requires-Dist: pytesseract; extra == "all"
Requires-Dist: pdf2image; extra == "all"
Requires-Dist: xlrd; extra == "all"
Requires-Dist: pyxlsb; extra == "all"
Requires-Dist: dbfread; extra == "all"
Requires-Dist: pyarrow; extra == "all"

# abstract_search

Standalone string search across files and directories with allow/exclude
filtering. Stdlib-only at import time — pdf, office, and spreadsheet readers
load lazily and are optional extras.

```python
from abstract_search import findContent, get_files_and_dirs, last_report

# 0.1.0: NO implicit filtering — searches EVERY readable file under the roots
hits = findContent("/path/to/project", strings=["needle"])

# opt into the old conveniences with a preset
hits = findContent("/path/to/project", strings=["needle"], preset="code")

# narrow explicitly; exclude a dir (exact segment, pruned)
hits = findContent("/path/to/project", strings=["needle"],
                   allowed_exts="py", exclude_dirs="tests")

# get the report inline (or call last_report())
res, rep = findContent("/path/to/project", strings=["needle"], report=True)
rep["effective_filters"]; rep["skipped"]
```

**NO implicit filtering (0.1.0).** With no filter kwargs you get EVERYTHING under
the roots — dotfiles, `node_modules`, `.venv`, binaries. There is no default
allow-whitelist, no default excluded dirs, no default hidden-file skipping.
`define_defaults` / `get_file_filters` reflect only user input + chosen presets.

Opt in explicitly with **presets** — `preset="code"` (`CODE_TEXT_EXTS` +
`EXCLUDE_NOISE_DIRS`, skip hidden), `preset="noise"` (`EXCLUDE_NOISE_DIRS` +
`BINARY_EXTS`, skip hidden), `preset="text"` — or pass the exported sets
`CODE_TEXT_EXTS` / `EXCLUDE_NOISE_DIRS` / `BINARY_EXTS`.

**Preset merge = UNION (never be surprised):** presets and your kwargs are
unioned field-by-field. Good for excludes; but the allow-whitelist is unioned
too, so a whitelist preset **broadens** — `preset="code"` + `allowed_exts=[".rs"]`
matches code/text AND `.rs`, not just `.rs`. To **narrow** by extension use
`preset="noise"` (excludes only) + your `allowed_exts`, or no preset.

When you do filter: exclude wins, include narrows, dir excludes are
exact path segments (pruned), globs match path AND basename, everything
case-insensitive. Content search skips binary files but **reports** them —
`abstract_search.engine.last_report()` (or `report=True`) carries the exact
`effective_filters` and `skipped` counts (`binary`/`unreadable`/`too_large`/
`symlink_loop`). The reader surface (`search_files`/`read_paths`, `WalkConfig`)
follows the same rule: `DEFAULT_WALK` is empty (searches everything); pass
`preset=`/filters to narrow; its report adds `extract_failed` for pdf/office read
errors. Remote/SSH search was removed in 0.1.0.

## Search engine v2 (2026-09-24)

The search core is an isolated, stdlib-only engine in `abstract_search/engine.py`
(a pruned `os.scandir` walk that replaced the old `find`-subprocess design). The
public modules (`filters`, `find_collect`, `find_content`) are thin compat layers
over it. Identify the live engine at runtime:

```python
import abstract_search
abstract_search.ENGINE      # "abstract_search.engine v2 (2026-09-24)"
```

A byte-for-byte sibling copy lives at `hugpy_tools.search` and is kept in step
(abstract_* never imports hugpy_*).

### Include / exclude semantics (now correct)
- **Exclude always wins; include narrows.** No include ⇒ everything not excluded.
- **Directory excludes** match an exact path **segment** and are **pruned** at
  walk time (an excluded tree is never entered) — `build` drops `.../build/...`
  but not `.../rebuild/...`.
- **Globs** (`allowed_patterns` / `exclude_patterns`) match the root-relative
  **path AND basename**, so `**/mct/*.py` and `*session*` work; exact names
  (`session.py`) match exactly.
- **Case:** names/globs/extensions are case-insensitive; literal content search
  is case-insensitive unless `case_sensitive=True`, and matches inside comments
  and URLs (the old normalizer dropped those).

`get_files_and_dirs(*args, recursive=True, include_files=True, **kwargs)` is the
most-used entry point; its `(dirs, files)` shape and kwargs are unchanged. See
its docstring for the full include/exclude contract and examples.

### Speed (best-of-2, 2026-09-24)

| tree | op | old (git HEAD) | engine v2 |
|---|---|---|---|
| `/srv/hugpy/src/hugpy` | `get_files_and_dirs` .py | 66.9 ms n=1046 | 19.4 ms n=1137 |
| | literal | 327 ms | 96 ms |
| | regex | 133 ms | 52 ms |
| `/srv/pyit/dev` | `get_files_and_dirs` .py | 366 ms n=4768 | 54 ms n=3351 |
| | literal | 933 ms | 132 ms |
| | regex | 775 ms | 130 ms |

Count differences are bug fixes: v2 finds the `__init__.py` files the old
default dropped (91 / 818), and old returned `.venv` / `site-packages` /
hidden-backup files (2235 on `/srv/pyit/dev`) that v2 prunes. v2's enumerate
counts match the `hugpy_tools.search` sibling exactly.

```python
from abstract_search import make_allowed_predicate, filter_allowed_items, search_files

allowed = make_allowed_predicate(exclude_exts=".log", add=True)
kept = filter_allowed_items(paths, allowed_exts=[".py", ".md"])
hits = search_files("needle", "/path/to/project")   # line-level SearchHit records
```

Optional readers: `pip install abstract_search[pdf]` (pdfplumber),
`[ocr]`, `[xls]`, `[xlsb]`, `[dbf]`, `[parquet]`, or `[all]`.
