Metadata-Version: 2.4
Name: polysearch-engine
Version: 0.1.1
Summary: Lightweight, pluggable hybrid (keyword + semantic) search over your own documents, backed by Postgres full-text search and Pinecone.
Author-email: Sulav <sulavthapachhetri@gmail.com>
License: MIT License
        
        Copyright (c) 2026 Sulav Thapa
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
Project-URL: Homepage, https://github.com/sulavchhetri/polysearch
Project-URL: Repository, https://github.com/sulavchhetri/polysearch
Project-URL: Issues, https://github.com/sulavchhetri/polysearch/issues
Keywords: search,hybrid-search,semantic-search,full-text-search,pinecone,rag
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: psycopg[binary]>=3.2
Requires-Dist: pinecone>=5.0
Requires-Dist: openai>=1.59
Provides-Extra: local
Requires-Dist: sentence-transformers>=3.0; extra == "local"
Provides-Extra: pdf
Requires-Dist: pypdf>=5.0; extra == "pdf"
Provides-Extra: docx
Requires-Dist: python-docx>=1.1; extra == "docx"
Provides-Extra: html
Requires-Dist: beautifulsoup4>=4.12; extra == "html"
Provides-Extra: pptx
Requires-Dist: python-pptx>=1.0; extra == "pptx"
Provides-Extra: ocr
Requires-Dist: rapidocr-onnxruntime>=1.3; extra == "ocr"
Requires-Dist: Pillow>=10.0; extra == "ocr"
Provides-Extra: db
Requires-Dist: SQLAlchemy>=2.0; extra == "db"
Provides-Extra: all
Requires-Dist: sentence-transformers>=3.0; extra == "all"
Requires-Dist: pypdf>=5.0; extra == "all"
Requires-Dist: python-docx>=1.1; extra == "all"
Requires-Dist: beautifulsoup4>=4.12; extra == "all"
Requires-Dist: python-pptx>=1.0; extra == "all"
Requires-Dist: rapidocr-onnxruntime>=1.3; extra == "all"
Requires-Dist: Pillow>=10.0; extra == "all"
Requires-Dist: SQLAlchemy>=2.0; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# polysearch

[![PyPI version](https://img.shields.io/pypi/v/polysearch-engine.svg)](https://pypi.org/project/polysearch-engine/)
[![Python versions](https://img.shields.io/pypi/pyversions/polysearch-engine.svg)](https://pypi.org/project/polysearch-engine/)
[![License](https://img.shields.io/pypi/l/polysearch-engine.svg)](https://github.com/sulavchhetri/polysearch/blob/main/LICENSE)

Lightweight, pluggable **hybrid (keyword + semantic) search** over your own documents.

polysearch pairs Postgres full-text search (the keyword engine) with Pinecone (the semantic
engine) and fuses their results with Reciprocal Rank Fusion. It is a plain Python **library and
CLI**: no web server, no ORM, no async, and connection details are passed in at runtime. Point it
at your Postgres and Pinecone, ingest files or database rows, and search.

> **Install name vs. import name:** the distribution on PyPI is **`polysearch-engine`**, but you
> import it as **`polysearch`**:
> ```bash
> pip install polysearch-engine
> ```
> ```python
> from polysearch import SearchService
> ```

## Features

- **Hybrid retrieval.** Keyword and semantic search run together and their rankings are merged, so
  exact-term lookups and natural-language questions both work well.
- **Keyword (sparse):** Postgres FTS with a graceful cascade, `phraseto_tsquery` then
  `websearch_to_tsquery` then a `:*` prefix fallback for search-as-you-type, ranked by `ts_rank`.
- **Semantic (dense):** OpenAI or a free local embedding model, stored and queried in Pinecone.
- **Fusion:** Reciprocal Rank Fusion (`K=60`) merges the two ranked lists. Each result is tagged
  `content` / `title` / `both`, with `matched_pages` (keyword hits) and `related_pages` (semantic hits).
- **Query intent:** short exact terms run keyword-only; natural-language questions add the semantic
  engine. The embedding model is never loaded for a keyword-only query.
- **Broad ingestion:** PDF, DOCX, HTML, PPTX, `.txt`, `.log`, images (via OCR or vision), and rows
  from an external database.
- **Multi-tenant:** every operation takes a `scope` (a namespace / tenant key) used as the Pinecone
  namespace and a Postgres filter.
- **Small core, opt-in extras:** the base install is just `psycopg` + `pinecone` + `openai`. The
  heavy dependencies (the local ML stack, file parsers, OCR) are optional extras you add only if you
  use them.

## How it works

```
                      query
                        │
             ┌──────────┴───────────┐
   keyword (Postgres FTS)   semantic (embeddings → Pinecone)
             └──────────┬───────────┘
              Reciprocal Rank Fusion (K=60)
                        │
                 ranked SearchResult[]
```

A query is classified as **keyword** or **semantic** with cheap string heuristics (no network).
Keyword queries hit Postgres only. Semantic queries also embed the query and search Pinecone, then
the two ranked lists are fused. Documents found by both engines rank highest.

## Requirements

- Python 3.10+
- A reachable **Postgres** database (the keyword / full-text store)
- A **Pinecone** index (the semantic / vector store), cloud or the local emulator
- Optionally an **OpenAI** API key for embeddings and image understanding

## Installation

The core stays small. Everything heavy is an opt-in extra:

```bash
pip install polysearch-engine                 # core
pip install 'polysearch-engine[local]'        # free local embeddings (sentence-transformers)
pip install 'polysearch-engine[pdf]'          # PDF (pypdf)
pip install 'polysearch-engine[docx]'         # DOCX (python-docx)
pip install 'polysearch-engine[html]'         # HTML (beautifulsoup4)
pip install 'polysearch-engine[pptx]'         # PPTX (python-pptx)
pip install 'polysearch-engine[ocr]'          # image OCR fallback (rapidocr-onnxruntime)
pip install 'polysearch-engine[db]'           # ingest non-Postgres external DBs (SQLAlchemy)
pip install 'polysearch-engine[all]'          # everything (convenience superset)
```

Install only the extras for the embedding backend and file types you actually use. The bulky
packages are **transitive**: `[local]` pulls the full ML stack (`torch`, `transformers`,
`scikit-learn`, and friends) via sentence-transformers, and `[ocr]` pulls `onnxruntime` / `opencv`
via rapidocr. Using OpenAI embeddings (no `[local]`) keeps the install small.

## Quickstart

You configure Postgres, Pinecone, and (optionally) OpenAI. On construction the service validates
each connection, ensures its schema, and selects the embedding backend from what validated:

- **OpenAI key present and valid** picks `text-embedding-3-small` (1536-dim), plus OpenAI vision for images.
- **No/invalid OpenAI key** picks a free local model (`BAAI/bge-base-en-v1.5`, 768-dim; needs `[local]`),
  plus RapidOCR for images (needs `[ocr]`).

```python
from polysearch import SearchService, PostgresConfig, PineconeConfig, OpenAIConfig

svc = SearchService(
    postgres=PostgresConfig(dsn="postgresql://user:pass@host:5432/db"),
    pinecone=PineconeConfig(api_key="...", index="polysearch", cloud="aws", region="us-east-1"),
    openai=OpenAIConfig(api_key="..."),   # optional; omit to use free local embeddings
)
print(svc.status())

svc.ingest_file("handbook.pdf", scope="docs")      # -> (file_id, chunks_indexed)
svc.ingest_path("./docs", scope="docs")            # a directory, ingested recursively

results = svc.search("database connection pooling", scope="docs")   # -> list[SearchResult]
for r in results:
    print(r.score, r.match_type, r.title, r.matched_pages, r.related_pages)

svc.delete("handbook", scope="docs")
svc.close()
```

`scope` is your namespace / tenant key. The same index can hold many scopes; searches and deletes
are always scoped.

The per-type ingestors are public too, if you only want to parse (no indexing):

```python
from polysearch import HTMLIngestor, DocxIngestor

parsed = HTMLIngestor().extract("page.html")   # -> ParsedDocument
```

## Search results

### As objects

`svc.search(query, scope)` returns `list[SearchResult]`. Each result has:

| field | meaning |
|---|---|
| `file_id` | stable id of the document |
| `title` | document title (may be `None`) |
| `score` | fused RRF score (higher is better) |
| `match_type` | where it matched: `content`, `title`, or `both` |
| `matched_pages` | pages with exact keyword hits ("found on page X") |
| `related_pages` | pages with semantic hits ("related content on page X") |

`svc.search_detailed(query, scope)` returns `(intent, results)` if you also want the classified
intent (`"keyword"` or `"semantic"`).

### As a JSON envelope

For a ready-to-serialize response, use `svc.search_json(query, scope)`:

```python
svc.search_json("how do I request VPN access?", scope="docs")
# {
#   "query": "how do I request VPN access?",
#   "scope": "docs",
#   "intent": "semantic",
#   "count": 2,
#   "results": [
#     {"file_id": "...", "title": "...", "score": 0.016,
#      "match_type": "content", "matched_pages": [], "related_pages": [1, 2]}
#   ]
# }
```

## CLI

The `polysearch` command mirrors the library. Flags fall back to environment variables and a
`.env` file (see [`.env.example`](.env.example)).

```bash
polysearch init                                   # validate config, report the chosen backend
polysearch status                                 # show enabled capabilities
polysearch ingest ./docs --scope docs             # a file or a directory
polysearch ingest handbook.pdf --scope docs --file-id handbook --title "Employee Handbook"
polysearch ingest-db --config db.json --scope docs
polysearch search "database connection pooling" --scope docs
polysearch delete handbook --scope docs
```

Every command accepts connection flags (`--database-url`, `--pinecone-api-key`, `--pinecone-index`,
`--pinecone-host`, `--openai-api-key`, `--embedding-backend`) and `-v` for verbose logging.

## Configuration

Config can be passed explicitly (the dataclasses above) or read from the environment (used by the
CLI). Defaults are shown below.

| Variable | Default | Purpose |
|---|---|---|
| `POLYSEARCH_DATABASE_URL` | (required) | Postgres DSN/URL for the keyword store (`DATABASE_URL` also accepted) |
| `PINECONE_API_KEY` | (required) | Pinecone API key |
| `PINECONE_INDEX` | `polysearch` | Pinecone index name |
| `PINECONE_CLOUD` | `aws` | Pinecone cloud (for index creation) |
| `PINECONE_REGION` | `us-east-1` | Pinecone region (`PINECONE_ENVIRONMENT` also accepted) |
| `PINECONE_HOST` | unset | Control-plane host for Pinecone Local / self-hosted; unset for cloud |
| `OPENAI_API_KEY` | unset | Optional; enables OpenAI embeddings and image vision |
| `POLYSEARCH_EMBEDDING_BACKEND` | `auto` | `auto` (OpenAI if valid, else local), `openai`, or `local` |
| `POLYSEARCH_LOCAL_MODEL` | `BAAI/bge-base-en-v1.5` | Local embedding model (needs `[local]`) |

## Ingesting an external database

Map a table's rows to documents (one row becomes one document). Postgres sources use the bundled
psycopg driver; other databases need `[db]` plus the matching driver (for example `pymysql`).

```json
{
  "dsn": "postgresql://user:pass@host:5432/src",
  "table": "articles",
  "id_column": "id",
  "text_columns": ["title", "body"],
  "title_column": "title",
  "scope_column": null,
  "batch_size": 200
}
```

```bash
polysearch ingest-db --config db.json --scope docs
```

## Embedding consistency and reindexing

Vectors from different embedding models are not comparable, so the service records which backend
built each Pinecone index (in a Postgres `embedding_state` row). If you later ingest or search with
a different backend it raises `EmbeddingMismatchError`. To switch backends, rebuild the dense side:

```bash
polysearch search "..." --scope docs --reindex     # CLI
```
```python
svc.reindex(force=True)                             # library
```

Reindex re-embeds the chunk text already stored in Postgres, so **no source files are needed**. The
keyword / FTS side is embedding-independent and untouched.

## Idempotency

Ingesting is idempotent per `(scope, file_id)`: re-ingesting a file replaces its chunks and reuses
its vector ids. The CLI derives a stable `file_id` from the file path unless you pass `--file-id`.

## Logging

polysearch is silent by default (it installs a null handler and does not touch your root logger).
Opt in from your app:

```python
import polysearch
polysearch.configure_logging("INFO")   # or "DEBUG"; omit the call to stay silent
```

The CLI is quiet unless you pass `-v`.

## Real-world example

A complete, runnable example lives in [`tests/e2e/`](tests/e2e/): it starts Postgres and Pinecone
Local in Docker, ingests sample PDF / PPTX / HTML / TXT / LOG files, and searches them using the
free local embedding backend (no OpenAI key). See [tests/e2e/README.md](tests/e2e/README.md).

## Development

```bash
pip install -e '.[all,dev]'
python -m pytest                     # unit tests (no network)
```

The full-text integration tests hit a real Postgres and are gated behind an env flag:

```bash
docker compose up -d db
RUN_DB_TESTS=1 DATABASE_URL=postgresql://search:search@localhost:5434/search \
    python -m pytest tests/test_integration_fts.py
```

## License

Released under the MIT License. See [LICENSE](LICENSE).
