Metadata-Version: 2.4
Name: needlesearch-local
Version: 0.1.1
Summary: A lightweight, local-first full-text search service.
Author: Noah Um
License-Expression: MIT
Project-URL: Homepage, https://github.com/fishfoodfish/NeedleSearch
Project-URL: Repository, https://github.com/fishfoodfish/NeedleSearch
Project-URL: Issues, https://github.com/fishfoodfish/NeedleSearch/issues
Keywords: search,search-engine,full-text-search,bm25,information-retrieval,local-first
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database :: Database Engines/Servers
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Needle Search

[![PyPI version](https://img.shields.io/pypi/v/needlesearch-local)](https://pypi.org/project/needlesearch-local/)
[![Python versions](https://img.shields.io/pypi/pyversions/needlesearch-local)](https://pypi.org/project/needlesearch-local/)
[![CI](https://github.com/fishfoodfish/NeedleSearch/actions/workflows/ci.yml/badge.svg)](https://github.com/fishfoodfish/NeedleSearch/actions/workflows/ci.yml)

Needle is a small full-text search engine built from scratch in Python. Give it
JSON documents, tell it which fields matter, and it will handle indexing,
ranking, filters, synonyms, and typo-tolerant search.

It is not tied to one kind of data. The same engine can search jobs, products,
recipes, articles, or anything else that can be represented as JSON.

```text
Search:  nike runing sneakers
                    ↓
Found:   Nike Pegasus Running Shoes
         Corrected "runing" to "running"
```

## Why I built it

Needle started as a way to improve search in Recruit Hub, but the interesting
part was bigger than job search. I wanted to understand what happens between a
user typing a query and receiving useful, ranked results.

So instead of hiding everything behind an existing search platform, Needle
implements the core pieces directly:

- an inverted index for fast text lookup;
- positional postings for phrase and proximity matching;
- BM25-style relevance ranking;
- different weights for important fields;
- context-aware typo correction with confidence scoring;
- bidirectional, multi-word synonyms, filters, and sorting;
- separate collections for unrelated datasets.

Python 3.11 or newer is all you need. There are no third-party dependencies.

## Installation

Install the latest release from [PyPI](https://pypi.org/project/needlesearch-local/):

```bash
python3 -m pip install needlesearch-local
```

Confirm that the command is available:

```bash
needle --version
```

To work on the project itself, clone the repository and install it in editable
mode:

```bash
git clone https://github.com/fishfoodfish/NeedleSearch.git
cd NeedleSearch
python3 -m pip install --editable .
```

## Architecture

```text
JSON documents
      │
      ▼
Schema validation
      │
      ▼
Text analysis and tokenization
      │
      ├──────────────► Trigram index ───► Context-aware typo correction
      │
      ▼
Inverted index
      │
      ▼
BM25 ranking + coverage + phrases + proximity + filters
      │
      ▼
CLI / HTTP API / browser
```

Each collection has its own schema, documents, index, synonyms, and autocorrect
vocabulary. Words learned from the `jobs` collection never leak into `products`
or `recipes`.

## Quick start

After installing Needle, initialize its local data and configuration:

```bash
needle init
```

Create a product collection using the example schema from this repository:

```bash
needle create products --schema schemas/products.json
```

Add the example products:

```bash
needle ingest examples/products.json --collection products
```

Now try a search—with a typo on purpose:

```bash
needle search "nike runing sneakers" --collection products
```

Needle corrects `runing`, expands `sneakers` using the collection's synonyms,
and ranks the Nike Pegasus result first.

For a stricter search, require a minimum number of query terms:

```bash
needle search "remote python internship" --collection jobs --minimum-should-match 2
```

By default, Needle keeps broad OR-style retrieval but reduces the score of
documents that match less of the query. `minimum_should_match` removes partial
matches entirely when an application needs more precision.

Use quotation marks when word order must be exact:

```bash
needle search '"software engineer"' --collection jobs
```

Unquoted terms can appear anywhere in a document, but terms that occur near one
another receive a proximity boost.

To see every collection:

```bash
needle collections
```

To run the local server:

```bash
needle serve --port 8080
```

Then open [http://127.0.0.1:8080](http://127.0.0.1:8080).
The current browser page searches the original default index; named collections
are available through the collection API shown below.

> If a collection already exists, you do not need to create it again. You can
> keep ingesting documents or search the existing collection.

## Measuring search quality

Needle can evaluate a collection against queries with human-assigned relevance
grades. Each line in a JSONL judgment file contains one query and the documents
that should match it:

```json
{"query":"amazon software internship","relevance":{"job-001":2}}
{"query":"python machine learning","relevance":{"job-003":2,"job-001":1}}
```

Use `2` for a highly relevant document, `1` for a partially relevant document,
and `0` for an irrelevant document. Typo examples can also include the
corrections Needle should make:

```json
{"query":"amazn intership","relevance":{"job-001":2},"corrections":{"amazn":"amazon","intership":"internship"}}
```

Evaluate the example jobs collection:

```bash
needle evaluate examples/jobs-judgments.jsonl --collection jobs
```

The report includes nDCG, mean reciprocal rank, recall, zero-result rate,
expected-no-result accuracy, correction accuracy, and average/P95 latency. Add
`--details` to inspect the returned document IDs and score for each query. Keep
the judgment file stable while refining the algorithm so improvements can be
compared against the same baseline.

The larger benchmark in `benchmarks/jobs/` contains 20 documents and 50
hand-labeled queries for comparing ranking changes.

### Local files

Needle keeps its indexes and configuration outside the installed Python package:

| Platform | Default data directory |
| --- | --- |
| macOS | `~/Library/Application Support/NeedleSearch/` |
| Linux | `~/.local/share/needlesearch/` |
| Windows | `%LOCALAPPDATA%\NeedleSearch\` |

Check the installation and local indexes with:

```bash
needle --version
needle doctor
```

Another project can keep its search data alongside the application:

```bash
needle --data-dir ./search-data serve
```

The same location can be set through the `NEEDLE_DATA_DIR` environment
variable. Use `NEEDLE_CONFIG_FILE` to override the configuration file.

## Try another kind of data

The workflow stays the same no matter what you are searching.

### Jobs

```bash
needle create jobs --schema schemas/jobs.json
needle ingest examples/jobs.json --collection jobs
needle search "amazn intership" --collection jobs
```

### Recipes

```bash
needle create recipes --schema schemas/recipes.json
needle ingest examples/recipes.json --collection recipes
needle search "spicy chiken" --collection recipes
```

## How a search works

Suppose you search for:

```text
amazn intership
```

Needle handles it in a few stages:

1. The query is normalized and split into `amazn` and `intership`.
2. Neither word exists in the collection's vocabulary, so Needle checks its
   trigram index for similar words.
3. Damerau-Levenshtein distance handles insertions, deletions, replacements,
   and adjacent transpositions.
4. Candidate confidence combines edit similarity, trigram overlap, term
   popularity, and overlap with the other query terms.
5. High-confidence corrected terms point directly to matching documents in the
   inverted index. Uncertain candidates are returned as suggestions without
   changing the search.
6. BM25 scores the results. Matches in important fields such as `title` and
   `company` receive a larger boost.
7. Query coverage reduces the score of documents that match only part of the
   query, and `minimum_should_match` can remove them entirely.
8. The highest-scoring documents are returned with the suggested corrections.

Correction responses distinguish applied corrections from uncertain
suggestions:

```json
{
  "corrections": {"amazn": "amazon"},
  "suggestions": {},
  "correction_details": {
    "amazn": {
      "term": "amazon",
      "confidence": 0.749,
      "applied": true
    }
  }
}
```

When two candidates are too close in confidence, Needle leaves the original
query unchanged and places the best candidate in `suggestions`.

The original query is still considered. Corrected words receive a smaller
ranking weight, so an exact match always has the advantage.

## Defining your own collection

A schema is a small JSON file that explains what your documents look like.
There is no need to change Needle's Python code.

Here is a shortened product schema:

```json
{
  "id_field": "sku",
  "fields": [
    {
      "name": "name",
      "type": "text",
      "searchable": true,
      "weight": 5
    },
    {
      "name": "brand",
      "type": "keyword",
      "searchable": true,
      "filterable": true,
      "weight": 2
    },
    {
      "name": "price",
      "type": "number",
      "searchable": false,
      "filterable": true,
      "sortable": true
    }
  ],
  "synonyms": {
    "swe": ["software engineer"],
    "intern": ["internship"]
  }
}
```

Synonyms are bidirectional. This configuration lets `swe` find `software
engineer`, `software engineer` find `swe`, and `internship` find documents that
use `intern`. A synonym value may contain one word or an entire phrase.

That schema accepts documents such as:

```json
{
  "sku": "shoe-934",
  "name": "Nike Pegasus Running Shoes",
  "brand": "Nike",
  "price": 129.99
}
```

Available field types:

| Type | Good for |
| --- | --- |
| `text` | Names, titles, descriptions, and other searchable writing |
| `text[]` | Ingredients, tags, skills, or any list of text |
| `keyword` | Brands, categories, or exact labels |
| `number` | Prices, salaries, ratings, and numeric ranges |
| `boolean` | Values such as `remote` or `in_stock` |
| `date` | ISO dates such as `2026-07-18` |

Needle validates every document before indexing it. If `price` is configured as
a number and a document sends `"cheap"`, ingestion fails with a clear error.

## Using the HTTP API

The CLI is convenient for experimenting. Applications such as Recruit Hub would
normally talk to Needle over HTTP.

Add documents:

```http
POST /v1/collections/products/documents
Content-Type: application/json
```

Search:

```http
POST /v1/collections/products/search
Content-Type: application/json

{
  "query": "nike runing shoes",
  "filters": {
    "price": {"lte": 150},
    "in_stock": true
  },
  "sort": [
    {"field": "_score", "order": "desc"}
  ],
  "limit": 20,
  "minimum_should_match": 2,
  "coverage_boost": 1.0
}
```

### API routes

| Method | Route | What it does |
| --- | --- | --- |
| `POST` | `/v1/collections` | Creates a collection |
| `GET` | `/v1/collections` | Lists existing collections |
| `GET` | `/v1/collections/{name}` | Shows its schema and statistics |
| `POST` | `/v1/collections/{name}/documents` | Adds one or more documents |
| `GET` | `/v1/collections/{name}/search?q=...` | Runs a simple search |
| `POST` | `/v1/collections/{name}/search` | Searches with filters and sorting |
| `DELETE` | `/v1/collections/{name}/documents/{id}` | Removes a document |
| `GET` | `/health` | Checks whether the server is running |

## Core files

| File | Purpose |
| --- | --- |
| `src/needlesearch/analysis.py` | Normalizes text, creates tokens and trigrams, and calculates bounded Damerau-Levenshtein distance |
| `src/needlesearch/engine.py` | Builds positional indexes, expands synonyms, finds corrections, matches phrases, filters documents, and ranks results |
| `src/needlesearch/evaluation.py` | Measures ranked relevance, corrections, zero results, and query latency |
| `src/needlesearch/models.py` | Defines schemas, field types, validation rules, and search results |
| `src/needlesearch/collections.py` | Keeps named collections and their indexes separate |
| `src/needlesearch/config.py` | Initializes configuration and checks local installation health |
| `src/needlesearch/paths.py` | Chooses safe data and configuration paths for each operating system |
| `src/needlesearch/server.py` | Exposes search through the browser and HTTP API |
| `src/needlesearch/cli.py` | Handles the `create`, `ingest`, `search`, and `serve` commands |
| `schemas/` | Example configurations for jobs, products, and recipes |
| `examples/` | Small datasets you can use while learning the project |
| `tests/` | Tests for indexing, ranking, corrections, collections, and the API |

## Project structure

```text
Search/
├── src/
│   └── needlesearch/
│       ├── analysis.py
│       ├── cli.py
│       ├── collections.py
│       ├── config.py
│       ├── engine.py
│       ├── evaluation.py
│       ├── models.py
│       ├── paths.py
│       └── server.py
├── .github/workflows/
│   ├── ci.yml
│   └── release.yml
├── schemas/
│   ├── jobs.json
│   ├── jobs-judgments.jsonl
│   ├── products.json
│   └── recipes.json
├── examples/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
└── tests/
    ├── test_collections.py
    ├── test_config.py
    ├── test_evaluation.py
    └── test_engine.py
```

## Running the tests

```bash
python3 -m unittest discover -s tests -v
```

The tests cover ranking, typo correction, typed validation, filtering, sorting,
persistence, isolated collection vocabularies, and the full HTTP lifecycle.

Every push and pull request runs the test suite on Python 3.11 through 3.14,
builds the wheel and source archive, validates the PyPI metadata, and installs
the wheel in a clean environment. Releases use PyPI Trusted Publishing, so the
repository does not store publishing tokens.

## Current scope

Needle is currently a single-node learning project. Text searches use posting
lists instead of scanning every document, but filters still scan document
metadata and saved indexes are rebuilt in memory when the engine starts.

The next meaningful improvements would be indexed filters, immutable disk
segments, a write-ahead log, and background compaction. Distributed search can
come later, once the single-node engine has real limits worth solving.

## License

Needle Search is available under the [MIT License](LICENSE).
