Metadata-Version: 2.4
Name: skeinrank
Version: 0.10.0
Summary: Dictionary-first extraction, alias canonicalization, and lightweight reranking utilities for technical documents.
License: Apache-2.0
License-File: LICENSE
Keywords: search,retrieval,reranking,rag,canonicalization,dictionary,aliases,documents
Author: NkvMax
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Linguistic
Requires-Dist: pydantic (>=2.6,<3)
Requires-Dist: typing-extensions (>=4.7) ; python_version < "3.11"
Project-URL: Homepage, https://github.com/SkeinRank
Project-URL: Issues, https://github.com/SkeinRank/skeinrank-core/issues
Project-URL: Repository, https://github.com/SkeinRank/skeinrank-core
Description-Content-Type: text/markdown

# skeinrank-core

`skeinrank-core` is the lightweight Python SDK and CLI for deterministic local terminology canonicalization.

It is the zero-friction entrypoint for SkeinRank: no Governance API, Elasticsearch, RabbitMQ, Celery, Docker, OpenRouter token, or ML dependencies are required.

## 30-second demo

```python
import skeinrank

print(skeinrank.canonicalize("k8s pg timeout"))
# kubernetes postgresql timeout

print(skeinrank.extract("sev1 on kube after deploy"))
# ['critical incident', 'kubernetes', 'deployment']
```

The module-level helpers use a built-in `platform_ops_demo` dictionary so the first call works without a file. The demo dictionary is small enough to inspect, but expressive enough to show infrastructure, incidents, CI/CD, search, RAG, and context-shaped company language.

The same built-in dictionary also demonstrates why context matters:

```python
import skeinrank

print(skeinrank.canonicalize("pg timeout"))
# postgresql timeout

print(skeinrank.canonicalize("pg layout"))
# page layout

print(skeinrank.canonicalize("pg dashboard"))
# product group
```

CLI from a source checkout:

```bash
poetry run skeinrank canonicalize "k8s pg timeout" --text
poetry run skeinrank extract "sev1 on kube after deploy" --text --compact
```

## Install from a checkout

```bash
cd packages/skeinrank-core
poetry install
poetry run pytest -q
```

Legacy reranking modules remain in the source tree for compatibility, but the package no longer exposes heavyweight ML install extras. The local SDK facade, demo dictionary, CLI, and document helpers do not require ML dependencies.

## Public Python facade

Use `SkeinRank` when you want to pass a dictionary in code:

```python
from skeinrank import SkeinRank

sr = SkeinRank({
    "kubernetes": ["k8s", "kube", "kuber"],
    "postgresql": ["pg", "postgres", "psql"],
})

print(sr.canonicalize("kuber timeout on pg"))
# kubernetes timeout on postgresql

print(sr.extract("kuber timeout on pg"))
# ['kubernetes', 'postgresql']
```

Use `explain=True` when you need offsets, slots, and highlighted evidence:

```python
result = sr.extract("k8s rollout uses pg", explain=True)

print(result.canonical_values)
print(result.matches[0].alias)
print(result.matches[0].highlighted_fragment)
```

The same facade can load a full SkeinRank dictionary JSON/YAML file:

```python
from skeinrank import SkeinRank

sr = SkeinRank.from_file("company.dictionary.yaml")
print(sr.canonicalize("k8s rollout uses pg database"))
```


## Built-in demo dictionary and examples

The built-in `platform_ops_demo` dictionary contains more than 30 canonical terms and more than 80 aliases across platform operations, incidents, CI/CD, search, RAG, and SkeinRank concepts. It is intentionally not a production vocabulary; it is a compact first-touch dictionary for demos, tests, tutorials, and screenshots.

Useful demo phrases:

| Input | Output |
| --- | --- |
| `k8s pg timeout` | `kubernetes postgresql timeout` |
| `sev1 on kube after pg migration` | `critical incident on kubernetes after postgresql database migration` |
| `gha deploy hit rmq latency spike` | `github actions`, `deployment`, `message queue`, `latency` |
| `pg layout` | `page layout` |
| `pg dashboard` | `product group` |

Examples live in [`../../examples/sdk`](../../examples/sdk):

- [`zero_friction_demo.py`](../../examples/sdk/zero_friction_demo.py) runs the facade from Python.
- [`platform_ops_demo.dictionary.json`](../../examples/sdk/platform_ops_demo.dictionary.json) exports the built-in dictionary in the public dictionary shape.

## Dictionary-first SDK

The lower-level dictionary SDK remains available for callers that already use the governance export or `skeinrank-migrate` dictionary shape.

```python
from skeinrank import load_dictionary, extract_terms, canonicalize_text

dictionary = load_dictionary("../../examples/migration/console_dictionary.example.json")

result = extract_terms(
    "This instruction helps deploy 500 k8s servers backed by Postgres.",
    dictionary=dictionary,
)

print(result.canonical_values)  # ['kubernetes', 'postgresql']

canonicalized = canonicalize_text(
    "k8s rollout uses pg database",
    dictionary=dictionary,
)
print(canonicalized.text)  # kubernetes rollout uses postgresql database
```

Stable dictionary exports include:

- `SkeinRank`, `canonicalize(...)`, `extract(...)`, `demo_dictionary(...)`, `demo_dictionary_payload(...)`
- `Dictionary`, `DictionaryTerm`, `DictionaryAlias`, `DictionaryStopListEntry`
- `load_dictionary(...)`, `validate_dictionary(...)`
- `extract_terms(...)`, `canonicalize_text(...)`
- `ExtractionResult`, `TermMatch`, `CanonicalizedText`

The matcher is deterministic and local. It honors active/deprecated term and alias statuses, profile/global stop lists, returns offsets, and includes evidence snippets with `<mark>...</mark>` highlights.

## Document text extraction utilities

Local document helpers can extract text before running the SDK matcher. They do not require the Governance API, Elasticsearch, Celery, or a database.

```python
from skeinrank import load_document_text, extract_terms_from_document

text = load_document_text("incident-runbook.md")
result = extract_terms_from_document(
    "incident-runbook.md",
    dictionary="../../examples/migration/console_dictionary.example.json",
)

print(result.document.file_name)
print(result.extraction.canonical_values)
```

Supported formats without extra dependencies:

- text-like files: `.txt`, `.md`, `.rst`, `.log`, `.csv`, `.tsv`, `.json`, `.jsonl`, `.yaml`, `.yml`
- `.html` / `.htm` with scripts/styles ignored
- `.docx` via a small stdlib ZIP/XML reader

PDF extraction is supported when the caller installs `pypdf` in the environment. The core package does not require it by default.

## Local CLI

Validate a dictionary exported from the governance API or used by `skeinrank-migrate`:

```bash
poetry run skeinrank validate-dictionary ../../examples/migration/console_dictionary.example.json
poetry run skeinrank validate-dictionary ../../examples/migration/console_dictionary.example.yaml --json
```

Run zero-config demo extraction/canonicalization:

```bash
poetry run skeinrank extract "k8s rollout uses pg database" --text --compact
poetry run skeinrank canonicalize "k8s rollout uses pg database" --text
```

Print or export the built-in demo dictionary:

```bash
poetry run skeinrank demo-dictionary --compact
poetry run skeinrank demo-dictionary --output ../../examples/sdk/platform_ops_demo.dictionary.json
```

Run the example script:

```bash
poetry run python ../../examples/sdk/zero_friction_demo.py
```

Run against a specific dictionary file:

```bash
poetry run skeinrank extract "k8s rollout uses pg database" \
  --text \
  --dictionary ../../examples/migration/console_dictionary.example.json

poetry run skeinrank canonicalize incident-runbook.md \
  --dictionary ../../examples/migration/console_dictionary.example.json \
  --output incident-runbook.canonicalized.txt
```

Extract plain text from a document before matching:

```bash
poetry run skeinrank document-text incident-runbook.docx --output incident-runbook.txt
```

The CLI returns JSON for `extract`, raw text by default for `canonicalize` and `document-text`, and supports `--output`, `--compact`, `--max-matches`, and `--context-chars` where relevant.

## Attribute extraction and enrichment

The older attribute/profile API is still available for advanced local enrichment workflows.

```python
from skeinrank import build_attribute_profile, enrich_texts

profile = build_attribute_profile(
    profile_id="company_terms",
    aliases={
        "kubernetes": ["k8s", "kube", "kuber"],
        "postgresql": ["pg", "postgres", "psql"],
    },
    slots={
        "kubernetes": "TOOL",
        "postgresql": "DB",
    },
    snapshot_version="company_terms@v1",
)

rows = enrich_texts(
    [
        {"id": "doc-1", "text": "k8s timeout after upgrade"},
        {"id": "doc-2", "text": "pg latency spike"},
    ],
    profile=profile,
)

print(rows[0]["canonical_values"])
```

Use this layer when you need profile templates, fuzzy alias fallback, richer passport/debug traces, or JSONL enrichment helpers.

## Publishing checklist

The package is published through the manual `publish-skeinrank-core` GitHub Actions workflow. The recommended flow is:

1. Build and test locally.
2. Publish to TestPyPI.
3. Install from TestPyPI in a clean environment.
4. Publish to PyPI only after the TestPyPI smoke test passes.

Local packaging checks:

```bash
poetry install
poetry run pytest -q
poetry build
poetry run python -m pip install --upgrade twine
poetry run twine check dist/*
```

See [`docs/PUBLISHING.md`](docs/PUBLISHING.md) for the full release checklist.

## Public API policy

Only symbols re-exported from `skeinrank.__init__` should be treated as stable public API. Internal modules may change without notice.

