Metadata-Version: 2.4
Name: needlesearch-local
Version: 0.1.0
Summary: A lightweight, local-first full-text search service.
Author: Noah Um
License-Expression: MIT
Project-URL: Homepage, https://github.com/fishfoodfish/NeedleSearch
Project-URL: Repository, https://github.com/fishfoodfish/NeedleSearch
Project-URL: Issues, https://github.com/fishfoodfish/NeedleSearch/issues
Keywords: search,search-engine,full-text-search,bm25,information-retrieval,local-first
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database :: Database Engines/Servers
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# Needle Search

Needle is a small full-text search engine built from scratch in Python. Give it
JSON documents, tell it which fields matter, and it will handle indexing,
ranking, filters, synonyms, and typo-tolerant search.

It is not tied to one kind of data. The same engine can search jobs, products,
recipes, articles, or anything else that can be represented as JSON.

```text
Search:  nike runing sneakers
                    ↓
Found:   Nike Pegasus Running Shoes
         Corrected "runing" to "running"
```

## Why I built it

Needle started as a way to improve search in Recruit Hub, but the interesting
part was bigger than job search. I wanted to understand what happens between a
user typing a query and receiving useful, ranked results.

So instead of hiding everything behind an existing search platform, Needle
implements the core pieces directly:

- an inverted index for fast text lookup;
- BM25-style relevance ranking;
- different weights for important fields;
- typo correction based on the indexed vocabulary;
- synonyms, filters, and sorting;
- separate collections for unrelated datasets.

Python 3.11 or newer is all you need. There are no third-party dependencies.

## Installation

Once the first release is published, install Needle with:

```bash
python3 -m pip install needlesearch-local
```

Until then, install the current repository locally:

```bash
python3 -m pip install .
```

## Architecture

```text
JSON documents
      │
      ▼
Schema validation
      │
      ▼
Text analysis and tokenization
      │
      ├──────────────► Trigram index ───► Typo correction
      │
      ▼
Inverted index
      │
      ▼
BM25 ranking + field weights + filters
      │
      ▼
CLI / HTTP API / browser
```

Each collection has its own schema, documents, index, synonyms, and autocorrect
vocabulary. Words learned from the `jobs` collection never leak into `products`
or `recipes`.

## Quick start

After installing Needle, initialize its local data and configuration:

```bash
needle init
```

Create a product collection using the example schema from this repository:

```bash
needle create products --schema schemas/products.json
```

Add the example products:

```bash
needle ingest examples/products.json --collection products
```

Now try a search—with a typo on purpose:

```bash
needle search "nike runing sneakers" --collection products
```

Needle corrects `runing`, expands `sneakers` using the collection's synonyms,
and ranks the Nike Pegasus result first.

To see every collection:

```bash
needle collections
```

To run the local server:

```bash
needle serve --port 8080
```

Then open [http://127.0.0.1:8080](http://127.0.0.1:8080).
The current browser page searches the original default index; named collections
are available through the collection API shown below.

> If a collection already exists, you do not need to create it again. You can
> keep ingesting documents or search the existing collection.

### Local files

Needle keeps its indexes and configuration outside the installed Python package:

| Platform | Default data directory |
| --- | --- |
| macOS | `~/Library/Application Support/NeedleSearch/` |
| Linux | `~/.local/share/needlesearch/` |
| Windows | `%LOCALAPPDATA%\NeedleSearch\` |

Check the installation and local indexes with:

```bash
needle --version
needle doctor
```

Another project can keep its search data alongside the application:

```bash
needle --data-dir ./search-data serve
```

The same location can be set through the `NEEDLE_DATA_DIR` environment
variable. Use `NEEDLE_CONFIG_FILE` to override the configuration file.

## Try another kind of data

The workflow stays the same no matter what you are searching.

### Jobs

```bash
needle create jobs --schema schemas/jobs.json
needle ingest examples/jobs.json --collection jobs
needle search "amazn intership" --collection jobs
```

### Recipes

```bash
needle create recipes --schema schemas/recipes.json
needle ingest examples/recipes.json --collection recipes
needle search "spicy chiken" --collection recipes
```

## How a search works

Suppose you search for:

```text
amazn intership
```

Needle handles it in a few stages:

1. The query is normalized and split into `amazn` and `intership`.
2. Neither word exists in the collection's vocabulary, so Needle checks its
   trigram index for similar words.
3. Edit distance confirms that `amazon` and `internship` are close matches.
4. The corrected terms point directly to matching documents in the inverted
   index.
5. BM25 scores the results. Matches in important fields such as `title` and
   `company` receive a larger boost.
6. The highest-scoring documents are returned with the suggested corrections.

The original query is still considered. Corrected words receive a smaller
ranking weight, so an exact match always has the advantage.

## Defining your own collection

A schema is a small JSON file that explains what your documents look like.
There is no need to change Needle's Python code.

Here is a shortened product schema:

```json
{
  "id_field": "sku",
  "fields": [
    {
      "name": "name",
      "type": "text",
      "searchable": true,
      "weight": 5
    },
    {
      "name": "brand",
      "type": "keyword",
      "searchable": true,
      "filterable": true,
      "weight": 2
    },
    {
      "name": "price",
      "type": "number",
      "searchable": false,
      "filterable": true,
      "sortable": true
    }
  ],
  "synonyms": {
    "sneakers": ["shoes", "trainers"]
  }
}
```

That schema accepts documents such as:

```json
{
  "sku": "shoe-934",
  "name": "Nike Pegasus Running Shoes",
  "brand": "Nike",
  "price": 129.99
}
```

Available field types:

| Type | Good for |
| --- | --- |
| `text` | Names, titles, descriptions, and other searchable writing |
| `text[]` | Ingredients, tags, skills, or any list of text |
| `keyword` | Brands, categories, or exact labels |
| `number` | Prices, salaries, ratings, and numeric ranges |
| `boolean` | Values such as `remote` or `in_stock` |
| `date` | ISO dates such as `2026-07-18` |

Needle validates every document before indexing it. If `price` is configured as
a number and a document sends `"cheap"`, ingestion fails with a clear error.

## Using the HTTP API

The CLI is convenient for experimenting. Applications such as Recruit Hub would
normally talk to Needle over HTTP.

Add documents:

```http
POST /v1/collections/products/documents
Content-Type: application/json
```

Search:

```http
POST /v1/collections/products/search
Content-Type: application/json

{
  "query": "nike runing shoes",
  "filters": {
    "price": {"lte": 150},
    "in_stock": true
  },
  "sort": [
    {"field": "_score", "order": "desc"}
  ],
  "limit": 20
}
```

### API routes

| Method | Route | What it does |
| --- | --- | --- |
| `POST` | `/v1/collections` | Creates a collection |
| `GET` | `/v1/collections` | Lists existing collections |
| `GET` | `/v1/collections/{name}` | Shows its schema and statistics |
| `POST` | `/v1/collections/{name}/documents` | Adds one or more documents |
| `GET` | `/v1/collections/{name}/search?q=...` | Runs a simple search |
| `POST` | `/v1/collections/{name}/search` | Searches with filters and sorting |
| `DELETE` | `/v1/collections/{name}/documents/{id}` | Removes a document |
| `GET` | `/health` | Checks whether the server is running |

## Core files

| File | Purpose |
| --- | --- |
| `src/needlesearch/analysis.py` | Normalizes text, creates tokens and trigrams, and calculates edit distance |
| `src/needlesearch/engine.py` | Builds indexes, finds corrections, filters documents, and ranks results |
| `src/needlesearch/models.py` | Defines schemas, field types, validation rules, and search results |
| `src/needlesearch/collections.py` | Keeps named collections and their indexes separate |
| `src/needlesearch/server.py` | Exposes search through the browser and HTTP API |
| `src/needlesearch/cli.py` | Handles the `create`, `ingest`, `search`, and `serve` commands |
| `schemas/` | Example configurations for jobs, products, and recipes |
| `examples/` | Small datasets you can use while learning the project |
| `tests/` | Tests for indexing, ranking, corrections, collections, and the API |

## Project structure

```text
Search/
├── src/
│   └── needlesearch/
│       ├── analysis.py
│       ├── cli.py
│       ├── collections.py
│       ├── engine.py
│       ├── models.py
│       └── server.py
├── schemas/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
├── examples/
│   ├── jobs.json
│   ├── products.json
│   └── recipes.json
└── tests/
    ├── test_collections.py
    └── test_engine.py
```

## Running the tests

```bash
python3 -m unittest discover -s tests -v
```

The tests cover ranking, typo correction, typed validation, filtering, sorting,
persistence, isolated collection vocabularies, and the full HTTP lifecycle.

Every push and pull request runs the test suite on Python 3.11 through 3.14,
builds the wheel and source archive, validates the PyPI metadata, and installs
the wheel in a clean environment. Releases use PyPI Trusted Publishing, so the
repository does not store publishing tokens.

## Current scope

Needle is currently a single-node learning project. Text searches use posting
lists instead of scanning every document, but filters still scan document
metadata and saved indexes are rebuilt in memory when the engine starts.

The next meaningful improvements would be indexed filters, immutable disk
segments, a write-ahead log, and background compaction. Distributed search can
come later, once the single-node engine has real limits worth solving.

## License

Needle Search is available under the [MIT License](LICENSE).
