Metadata-Version: 2.4
Name: synap-er
Version: 0.4.0
Summary: B2B entity resolution engine built on advanced-text-processing
Author: Gaurav Dadhich
License: MIT
Project-URL: Repository, https://github.com/maximem-ai/synap-er
Project-URL: Issues, https://github.com/maximem-ai/synap-er/issues
Keywords: entity-resolution,ner,b2b,canonicalization,knowledge-graph,registry
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pydantic-settings>=2.0
Requires-Dist: rapidfuzz>=3.0.0
Requires-Dist: jellyfish>=1.0.0
Requires-Dist: faiss-cpu>=1.7.4
Requires-Dist: pybktree>=1.1
Requires-Dist: python-Levenshtein>=0.21.0
Requires-Dist: spacy>=3.7.0
Requires-Dist: sentence-transformers>=2.2.0
Requires-Dist: requests>=2.31.0
Requires-Dist: cachetools>=5.3.0
Requires-Dist: flashtext>=2.7
Requires-Dist: numpy>=1.24.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Provides-Extra: ner
Requires-Dist: transformers>=4.30.0; extra == "ner"
Requires-Dist: spacy-wordnet>=0.1.0; extra == "ner"
Requires-Dist: nltk<3.6,>=3.3; extra == "ner"
Requires-Dist: requests-ratelimiter>=0.4.0; extra == "ner"
Requires-Dist: hnswlib>=0.7.0; extra == "ner"
Requires-Dist: dedupe>=2.0.0; extra == "ner"
Requires-Dist: recordlinkage>=0.15.0; extra == "ner"
Requires-Dist: autofaiss>=2.17.0; extra == "ner"
Requires-Dist: pandas>=2.0.0; extra == "ner"
Provides-Extra: llm
Requires-Dist: litellm>=1.0.0; extra == "llm"
Provides-Extra: dev
Requires-Dist: pytest>=7.4.0; extra == "dev"
Requires-Dist: pytest-cov>=4.1.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: httpx>=0.24.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: flake8>=6.0.0; extra == "dev"
Requires-Dist: mypy>=1.4.0; extra == "dev"
Requires-Dist: isort>=5.12.0; extra == "dev"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=1.3.0; extra == "docs"
Requires-Dist: myst-parser>=2.0.0; extra == "docs"
Dynamic: license-file

# synap-er

[![Python Versions](https://img.shields.io/pypi/pyversions/advanced-text-processing.svg)](https://pypi.org/project/advanced-text-processing/)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)

## Overview

synap-er is a B2B entity resolution library that extends [advanced-text-processing](https://github.com/dadhichgaurav1/advanced-text-processing) (`ner_lib`) with scoped registries, a multi-stage resolution pipeline, and in-memory indexing for sub-millisecond lookups. It is designed to be imported as a library into a host application — it is **not** a standalone microservice.

Key capabilities:

- **Multi-level scoped registries** (World → Client → Customer → User) with confidence-weighted inheritance
- **6-stage resolution pipeline** (exact → fuzzy → semantic → contextual → LLM-optional → manual review)
- **Write-through consistency:** PostgreSQL as durable store, in-memory indices for runtime reads
- **Corpus adapters** for domain-specific data (UNSPSC, UMLS, schema.org, etc.)
- **Optional vendor integration** (Senzing, Zingg) for clients with existing MDM infrastructure
- **No GPU required** — all components run on CPU

## Installation

### From source (development)

```bash
git clone git@github.com:gauravmaximem/synap-er.git
cd synap-er
pip install -e ".[dev]"
```

### As a dependency in another project

**Option A: Git SSH (for development machines)**

```
# In your host project's requirements.txt
synap-er @ git+ssh://git@github.com/gauravmaximem/synap-er.git@v0.1.0
```

**Option B: Git HTTPS with token (for CI/CD and deployment)**

```
# In your host project's requirements.txt
synap-er @ git+https://${GITHUB_TOKEN}@github.com/gauravmaximem/synap-er.git@v0.1.0
```

**Option C: Vendored wheel (zero auth required)**

```bash
# Download the wheel from a GitHub Release
gh release download v0.1.0 --repo gauravmaximem/synap-er --pattern "*.whl" --dir vendor/

# In your host project's requirements.txt
./vendor/synap_er-0.1.0-py3-none-any.whl
```

### CI/CD Setup (GitHub Actions)

For Option B, add this step before `pip install`:

```yaml
- name: Configure git for private repo access
  run: git config --global url."https://${GITHUB_TOKEN}@github.com/".insteadOf "https://github.com/"
  env:
    GITHUB_TOKEN: ${{ secrets.SYNAP_ER_TOKEN }}
```

For Option C, no auth is needed — the wheel is committed to your repo.

## The registry is yours

synap-er resolves names against a registry of canonical entities. It does not
own one. It opens no database connection, creates no tables and writes no rows.
Your application decides where entries live, how they are written and how a
change reaches every worker. The library says what it needs, in
`synap_er.registry`:

| | What it is |
|---|---|
| `RegistryEntryLike` | The attributes the resolver reads from an entry: `canonical_id`, `canonical_name`, `entity_type`, `scope`, `scope_id`, `aliases`, `metadata_`, `embedding`. Any object that has them will do, including your own ORM rows. |
| `RegistryEntry` | A plain dataclass with those attributes, for hosts that do not need a model of their own. |
| `Registry` | The lookups the resolver makes. |
| `ReviewSink` | Where the threshold path hands a mention it could not place. Optional. |
| `EntityType`, `Scope`, `Source` | The enums an entry is described with. |

`synap_er.memory.RegistryIndexManager` is an in-memory registry that satisfies
`Registry`. The usual arrangement is:

1. Keep entries in your own database, in whatever shape you like.
2. At start-up, load them and call `index.build(entries)`.
3. On every write, update the index too: `add_entry`, `update_entry`,
   `remove_entry`. If you run several workers, tell the others.

An entry's `embedding` is a 384-float sentence embedding of its name
(`synap_er.memory.compute_embedding`). Entries without one are found by name
only, not by semantic search.

## Configuration

Resolution settings come from environment variables (prefix `SYNAP_ER_`) or
from code:

```python
from synap_er.config import PipelineSettings, Settings

settings = Settings(pipeline=PipelineSettings(confidence_threshold=0.85))
```

## Usage

```python
from synap_er.config import Settings
from synap_er.memory import RegistryIndexManager
from synap_er.pipeline.adjudication import MentionInput
from synap_er.pipeline.resolver import PipelineResolver
from synap_er.registry import EntityType, RegistryEntry

# Start-up: your entries, from your storage, into the index.
index = RegistryIndexManager()
index.build([
    RegistryEntry("Acme Corporation", EntityType.ORG, "CLIENT", "client-123",
                  aliases=["Acme Corp", "ACME"]),
])
resolver = PipelineResolver(index, Settings())

# One call for every name in a document. Narrowest scope first.
scope_chain = [("CLIENT", "client-123"), ("WORLD", None)]
[result] = await resolver.resolve_many(
    [MentionInput("Acme Corp", EntityType.ORG)], scope_chain,
    context="Acme Corp is renewing the contract.",
)
result.decision        # MATCH, NEW or AMBIGUOUS
result.canonical_name  # "Acme Corporation"
```

What you do with the answer is yours as well: on `NEW`, write an entry to your
storage and `index.add_entry(...)`; on `AMBIGUOUS`, queue it for a person.
Pass `adjudicator=` to have doubtful pairs judged; without one they are
decided by score.

### Using the original ner_lib (world-data canonicalization)

The full ner_lib library is bundled and works unchanged:

```python
from ner_lib import canonicalize_entity, canonicalize_relationship

result = canonicalize_entity("Apple Inc", mode="progressive")
print(result["canonical_name"])  # "Apple Inc."
```

## Development & Testing

```bash
pytest tests/synap_er/ -v
```

The tests need no database: the library has no storage.

## Architecture

See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed design decisions.

Key points:

- **Library, not a service.** synap-er is imported into your application process. No separate deployment.
- **Your database, not ours.** synap-er adds tables to the host application's PostgreSQL instance.
- **Write-through consistency.** PostgreSQL is the durable store. In-memory structures (hash maps, BK-trees, FAISS, PhraseMatcher) serve all runtime reads. Updates are synchronous — a write is visible immediately.
- **No GPU required.** All components (spaCy, sentence-transformers, FAISS, RapidFuzz) run on CPU at the projected registry sizes.

## Upstream Sync

synap-er is a private extension of the public [advanced-text-processing](https://github.com/dadhichgaurav1/advanced-text-processing) library. To pull in updates from the public repo:

```bash
git fetch upstream
git merge upstream/main
```

The `ner_lib/` directory is upstream's code. The `synap_er/` directory is the B2B extension. They never conflict because synap_er imports from ner_lib but never modifies it.

## License

Proprietary. All rights reserved.
