Metadata-Version: 2.4
Name: searchbox
Version: 0.2.0
Summary: A lightweight local document ingestion and retrieval package.
Author: searchbox
Keywords: search,retrieval,documents,sqlite,pdf
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# searchbox

`searchbox` is a lightweight Python package for local document ingestion and retrieval.

It does one thing:

- read local files or folders
- extract text from supported document types
- split long text into chunks
- store the chunks in SQLite
- retrieve the most relevant documents for a query

## Install

Install from the current project directory:

```bash
pip install .
```

If you later publish it to PyPI under the same package name:

```bash
pip install searchbox
```

## What It Supports

Supported file types:

- `.txt`, `.md`, `.rst`
- `.py`, `.toml`, `.yaml`, `.yml`
- `.csv`, `.tsv`
- `.json`
- `.html`, `.htm`
- `.pdf`

Storage backend:

- SQLite database file on local disk

Retrieval behavior:

- documents are stored as chunks internally
- search is performed on chunks
- results are returned as document-level hits
- each result includes the best matching chunk text and matched chunk indexes

## Quick Start

```python
from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)
engine.add_path("docs")

results = engine.search("how does indexing work", top_k=3)

for item in results:
    print(item["id"], item["score"], item["metadata"]["matched_chunks"])
```

## Main API

Create a search engine:

```python
from searchbox import docsearch

engine = docsearch(
    db_path="docs.db",
    chunk_size=240,
    chunk_overlap=40,
)
```

Available methods:

- `docsearch(db_path="searchbox.db", chunk_size=240, chunk_overlap=40)`
- `add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)`
- `add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)`
- `search(query, top_k=5)`
- `search_pretty(query, top_k=5, snippet_length=280)`
- `response(query, top_k=5)`
- `count()`
- `clear()`
- `supported_extensions()`

## Constructor Parameters

### `docsearch(...)`

```python
engine = docsearch(
    db_path="searchbox.db",
    chunk_size=240,
    chunk_overlap=40,
)
```

Parameters:

- `db_path`: SQLite database path; default is `searchbox.db`
- `chunk_size`: number of words per chunk; default is `240`
- `chunk_overlap`: overlapping words between neighboring chunks; default is `40`

Notes:

- `chunk_size` must be greater than `0`
- `chunk_overlap` must be greater than or equal to `0`
- `chunk_overlap` must be smaller than `chunk_size`

## Ingestion Methods

### `add_path(path, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)`

Index one file or one directory.

If `path` is:

- a file: only that file is indexed
- a directory: files inside the directory are indexed

Returns:

- a list of inserted chunk ids

Parameters:

- `path`: file path or directory path
- `recursive`: when `path` is a directory, whether to scan subdirectories; default `True`
- `include_hidden`: whether to include hidden files and hidden directories; default `False`
- `chunk_size`: optional per-call override of constructor `chunk_size`
- `chunk_overlap`: optional per-call override of constructor `chunk_overlap`

Example:

```python
engine.add_path("docs")
engine.add_path("README.md")
engine.add_path("docs", recursive=False)
```

### `add_paths(paths, recursive=True, include_hidden=False, chunk_size=None, chunk_overlap=None)`

Index multiple files or directories in one call.

Parameters:

- `paths`: a sequence of file or directory paths
- all other parameters behave the same as `add_path(...)`

Example:

```python
engine.add_paths(
    [
        "docs",
        "notes/project.md",
        "data/reference",
    ]
)
```

## Search Methods

### `search(query, top_k=5)`

Search the indexed documents and return the most relevant document hits.

Parameters:

- `query`: search text
- `top_k`: maximum number of returned documents; default `5`

Returns:

- a list of result dictionaries sorted by descending relevance

### `response(query, top_k=5)`

Alias of `search(query, top_k=5)`.

Use whichever name you prefer.

### `search_pretty(query, top_k=5, snippet_length=280)`

Search the indexed documents and return a more display-friendly result shape.

Parameters:

- `query`: search text
- `top_k`: maximum number of returned documents; default `5`
- `snippet_length`: max snippet length in characters; default `280`

Returns:

- a list of simplified dictionaries with `path`, `file_name`, `score`, `matched_chunk_count`, `matched_chunks`, `best_chunk_index`, and `snippet`

## Examples

### Index a single file

```python
from searchbox import docsearch

engine = docsearch(db_path="single.db")
engine.add_path("guide.txt")

results = engine.search("installation guide", top_k=1)
print(results)
```

### Index a directory

```python
from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("docs")

results = engine.search("local retrieval", top_k=3)
for item in results:
    print(item["id"], item["title"], item["score"])
```

### Index multiple locations

```python
from searchbox import docsearch

engine = docsearch(db_path="multi.db")
engine.add_paths(["docs", "notes", "README.md"])

results = engine.search("api design", top_k=5)
for item in results:
    print(item["metadata"]["source_path"])
```

### Override chunk settings for one ingestion call

```python
from searchbox import docsearch

engine = docsearch(db_path="docs.db", chunk_size=240, chunk_overlap=40)

engine.add_path(
    "manuals",
    chunk_size=120,
    chunk_overlap=20,
)
```

### Show supported file types

```python
from searchbox import docsearch

engine = docsearch()
print(engine.supported_extensions())
```

### Get display-friendly search output

```python
from searchbox import docsearch

engine = docsearch(db_path="docs.db")
engine.add_path("papers")

results = engine.search_pretty("vllm", top_k=3)

for item in results:
    print(item["path"])
    print(item["score"])
    print(item["matched_chunk_count"])
    print(item["snippet"])
    print("---")
```

## Search Result Format

Each search result is a document-level hit.

Example:

```python
[
    {
        "id": "guide.pdf",
        "title": "guide.pdf",
        "content": "best matching chunk text ...",
        "score": 0.82,
        "metadata": {
            "source_path": "guide.pdf",
            "source_name": "guide.pdf",
            "chunk_count": 4,
            "matched_chunks": [0, 1],
            "best_chunk_index": 1,
        },
    }
]
```

Field meaning:

- `id`: document identifier; currently the source file path string used by the result layer
- `title`: display title derived from the file name
- `content`: the best matching chunk text for that document
- `score`: relevance score
- `metadata["source_path"]`: file path stored for the source document
- `metadata["source_name"]`: file name only
- `metadata["chunk_count"]`: total number of stored chunks for the file
- `metadata["matched_chunks"]`: chunk indexes that matched the query
- `metadata["best_chunk_index"]`: chunk index whose text is returned in `content`

## How Ingestion Works

When you call `add_path(...)` or `add_paths(...)`:

1. `searchbox` reads each supported file
2. extracted text is normalized
3. long text is split into overlapping word chunks
4. every chunk is stored in the SQLite database
5. later, `search(...)` ranks chunks and merges them back into document-level results

This means:

- large files can still be searched effectively
- results are returned as documents, not raw chunks
- `content` in the result is the best matching chunk, not always the full original file

## PDF Notes

PDF ingestion is supported.

Current behavior:

- `searchbox` first tries the system command `pdftotext`
- if that fails, it uses a lightweight fallback extractor

Recommendation:

- if you care about PDF extraction quality, make sure `pdftotext` is installed on the machine

## Utility Methods

### `count()`

Return the number of stored chunk rows in the SQLite database.

Example:

```python
count = engine.count()
print(count)
```

### `clear()`

Delete all stored chunk rows from the database.

Example:

```python
engine.clear()
```

### `supported_extensions()`

Return the currently supported file extensions.

Example:

```python
print(engine.supported_extensions())
```

## Behavior and Edge Cases

- unsupported file types are skipped
- empty or unreadable extracted content is skipped
- hidden files are skipped by default when scanning directories
- if a given path does not exist, `FileNotFoundError` is raised
- if `query` is empty or blank, `ValueError` is raised
- if no document matches, `search(...)` returns an empty list

## Minimal End-to-End Example

```python
from searchbox import docsearch

engine = docsearch(db_path="example.db")

engine.add_paths(
    [
        "docs",
        "README.md",
        "notes/plan.json",
    ]
)

results = engine.search("sqlite document retrieval", top_k=3)

for item in results:
    print("path:", item["metadata"]["source_path"])
    print("score:", item["score"])
    print("best chunk:", item["content"])
    print("matched chunks:", item["metadata"]["matched_chunks"])
    print("---")
```
