Metadata-Version: 2.4
Name: nlp4j-local-search-embedding
Version: 0.3.0
Summary: Embedding-based semantic search built on nlp4j-local-search
Author: Hiroki Oya
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/oyahiroki/nlp4j-local-search-embedding
Project-URL: Repository, https://github.com/oyahiroki/nlp4j-local-search-embedding
Project-URL: Issues, https://github.com/oyahiroki/nlp4j-local-search-embedding/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: nlp4j-local-search>=0.3.0
Requires-Dist: sentence-transformers>=2.7.0
Dynamic: license-file

# nlp4j-local-search-embedding

`nlp4j-local-search-embedding` provides simple embedding-based semantic search built on top of [`nlp4j-local-search`](https://github.com/oyahiroki/nlp4j-local-search).

It converts text into embeddings using a multilingual E5 model and stores the resulting vectors in a local vector search index.

## Features

* Simple semantic search API
* Text-to-vector embedding with multilingual E5
* Local vector search using `nlp4j-local-search`
* E5 query/passage prefixes handled internally
* Supports document IDs, text, and metadata
* **Field search** — filter search results by document fields (added in v0.3.0)
* Designed as a lightweight bridge between local search and embedding models

## Installation

`nlp4j-local-search-embedding` is available on PyPI.

Install the latest version with:

```bash
pip install nlp4j-local-search-embedding==0.3.0
```

This package depends on `nlp4j-local-search` and `sentence-transformers`.

The first run may take some time because the embedding model is downloaded and loaded locally.

## Local Development Installation

For local development with editable installs:

```bash
git clone https://github.com/oyahiroki/nlp4j-local-search.git
git clone https://github.com/oyahiroki/nlp4j-local-search-embedding.git

python -m venv .venv
source .venv/bin/activate

python -m pip install -U pip setuptools wheel
python -m pip install -e ./nlp4j-local-search
python -m pip install -e ./nlp4j-local-search-embedding
```

On Windows PowerShell:

```powershell
git clone https://github.com/oyahiroki/nlp4j-local-search.git
git clone https://github.com/oyahiroki/nlp4j-local-search-embedding.git

python -m venv .venv
.\.venv\Scripts\Activate.ps1

python -m pip install -U pip setuptools wheel
python -m pip install -e ./nlp4j-local-search
python -m pip install -e ./nlp4j-local-search-embedding
```

## Quick Start

```python
from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add({
    "doc1": "Kyoto is a historic city in Japan.",
    "doc2": "Tokyo is the capital city of Japan.",
    "doc3": "Python is a popular programming language.",
})

app.commit()

results = app.search("an old Japanese capital", limit=10)

print("=== Search results ===")
print(f"number of results: {len(results)}")

for i, result in enumerate(results):
    print(f"result[{i}].id: {result.id}")
    print(f"result[{i}].text: {result.text}")
    print(f"result[{i}].score: {result.score}")
    print(f"result[{i}].metadata: {result.metadata}")
    print("---")
```

Example output:

```text
=== Search results ===
number of results: 3
result[0].id: doc1
result[0].text: Kyoto is a historic city in Japan.
result[0].score: 0.91
result[0].metadata: {}
---
result[1].id: doc2
result[1].text: Tokyo is the capital city of Japan.
result[1].score: 0.88
result[1].metadata: {}
---
result[2].id: doc3
result[2].text: Python is a popular programming language.
result[2].score: 0.75
result[2].metadata: {}
---
```

Scores may vary depending on the model version and runtime environment.

## Using Metadata

Documents can include metadata.

```python
from nlp4j_local_search_embedding import SemanticSearch

documents = [
    {
        "id": "doc1",
        "text": "Kyoto is a historic city in Japan.",
        "metadata": {
            "category": "city",
            "country": "Japan"
        },
    },
    {
        "id": "doc2",
        "text": "Nintendo is a video game company headquartered in Kyoto.",
        "metadata": {
            "category": "company",
            "country": "Japan"
        },
    },
    {
        "id": "doc3",
        "text": "Python is widely used for data science and machine learning.",
        "metadata": {
            "category": "technology"
        },
    },
]

app = SemanticSearch("en")
app.add(documents)
app.commit()

results = app.search("a Japanese game company", limit=3)

for result in results:
    print(result.id)
    print(result.text)
    print(result.score)
    print(result.metadata)
    print("---")
```

## Adding a Single Document

```python
from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add(
    "doc1",
    "Kyoto is known for temples, shrines, and traditional culture.",
    metadata={"category": "travel"}
)

app.commit()

results = app.search("traditional Japanese culture", limit=5)

for result in results:
    print(result.id, result.score, result.text)
```

## Field Search

Added in v0.3.0.

Documents can be registered with **fields** — key-value pairs used for exact-match filtering.
Use `fields=` when adding documents and `filters=` when searching.

`filters` conditions are combined with AND when multiple fields are specified.
Field values are evaluated by exact (term) match and do not affect the similarity score.

### Registering documents with fields

```python
from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch("en")

app.add([
    {"id": "1", "text": "Kyoto is a historic city in Japan.",
     "fields": {"category": "city", "country": "Japan"}},
    {"id": "2", "text": "Nintendo is headquartered in Kyoto, Japan.",
     "fields": {"category": "company", "country": "Japan"}},
    {"id": "3", "text": "Tokyo is the capital city of Japan.",
     "fields": {"category": "city", "country": "Japan"}},
    {"id": "4", "text": "Paris is the capital city of France.",
     "fields": {"category": "city", "country": "France"}},
    {"id": "5", "text": "Sony is a Japanese multinational company.",
     "fields": {"category": "company", "country": "Japan"}},
])
app.commit()
```

### Filtering by a single field

```python
results = app.search("", limit=10, filters={"category": "city"})
for r in results:
    print(r.id, r.text)
# [1] Kyoto is a historic city in Japan.
# [3] Tokyo is the capital city of Japan.
# [4] Paris is the capital city of France.
```

### Filtering by multiple fields (AND)

```python
results = app.search("", limit=10, filters={"category": "city", "country": "Japan"})
for r in results:
    print(r.id, r.text)
# [1] Kyoto is a historic city in Japan.
# [3] Tokyo is the capital city of Japan.
```

### Semantic search combined with field filtering

```python
results = app.search("Japanese company", limit=10, filters={"category": "company"})
for r in results:
    print(r.id, r.score, r.text)
# Only documents with category="company" are returned, ranked by semantic similarity.
```

### Two-argument form with fields

```python
app.add("doc1", "Kyoto is a historic city in Japan.",
        fields={"category": "city", "country": "Japan"})
```

### Using the Document class with fields

```python
from nlp4j_local_search_embedding import Document, SemanticSearch

app = SemanticSearch("en")
app.add([
    Document(id="1", text="Kyoto is a historic city in Japan.",
             fields={"category": "city", "country": "Japan"}),
    Document(id="2", text="Nintendo is headquartered in Kyoto, Japan.",
             fields={"category": "company", "country": "Japan"}),
])
app.commit()

results = app.search("old capital", limit=5, filters={"category": "city"})
```

## Default Model

The default embedding model is:

```text
intfloat/multilingual-e5-large
```

`SemanticSearch` uses the E5-style prefixes internally:

```text
passage: <document text>
query:   <search query>
```

Therefore, users can simply add plain document text and search with plain query text.

## Specifying a Model

```python
from nlp4j_local_search_embedding import SemanticSearch

app = SemanticSearch(
    "en",
    model_name="intfloat/multilingual-e5-large"
)
```

## Architecture

This package is designed as an embedding layer for `nlp4j-local-search`.

```text
text documents
    |
    v
E5 embedding model
    |
    v
vectors
    |
    v
nlp4j-local-search vector index
    |
    v
semantic search results
```

The base package `nlp4j-local-search` is responsible for local search and vector indexing.
This package is responsible for converting text into embeddings and providing a convenient `SemanticSearch` API.

## Relationship with nlp4j-local-search

`nlp4j-local-search` can perform keyword search and vector search with user-provided vectors.

```python
from nlp4j_local_search import SearchEngine

search = SearchEngine("en", vector_dimension=2)
search.add("east", [1.0, 0.0])
search.add("north", [0.0, 1.0])
search.commit()

results = search.search([0.9, 0.1], limit=10)
```

`nlp4j-local-search-embedding` adds text embedding support on top of that.

```python
from nlp4j_local_search_embedding import SemanticSearch

search = SemanticSearch("en")
search.add({
    "doc1": "Kyoto is a historic city in Japan.",
    "doc2": "Tokyo is the capital city of Japan.",
})
search.commit()

results = search.search("old Japanese capital", limit=10)
```

## Saving Document Text and Metadata

The vector index is managed by `nlp4j-local-search`.

This package also keeps document text and metadata on the Python side so that search results can include the original text.

```python
app.save_documents("documents.json")
```

To restore the document text and metadata:

```python
app.load_documents("documents.json")
```

Note: index persistence and document-store persistence may be handled separately depending on the version of `nlp4j-local-search`.

## Development

Install the package in editable mode:

```bash
python -m pip install -e .
```

Run an example:

```bash
python examples/simple_semantic_search.py
```

Run tests:

```bash
python -m pip install pytest
python -m pytest
```

## Project Structure

```text
nlp4j-local-search-embedding/
  src/
    nlp4j_local_search_embedding/
      __init__.py
      document.py
      e5_embedder.py
      errors.py
      result.py
      semantic_search.py
  examples/
    simple_semantic_search.py
    example_0.3.0.py
  tests/
  pyproject.toml
  README.md
  README_build.md
  LICENSE
```

## Notes

* The first run may take time because the embedding model needs to be downloaded and loaded.
* The package depends on `sentence-transformers`.
* The base vector search functionality is provided by `nlp4j-local-search`.
* This package is intended for local semantic search, experimentation, and lightweight RAG-style applications.

## License

Apache License 2.0

