Metadata-Version: 2.4
Name: categorizer
Version: 2.0.0
Summary: LLM based multi-lvl text categorization tool
Author: Enes Kuzucu
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: pydantic>=2.7
Requires-Dist: PyYAML>=6.0
Requires-Dist: langchain-openai>=0.3
Requires-Dist: python-dotenv>=1.0
Provides-Extra: pandas
Requires-Dist: pandas>=2.0; extra == "pandas"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Categorizer

Easily categorize string records into predefined categories with a
combination of LLM, regex and keywords.

- Define your categories, feed your input — records come back categorized.
- Nested categories are allowed (any depth).
- For efficiency, regex patterns and keywords can pre-handle records where
  applicable, and the LLM acts as the fallback — you only pay for what rules
  can't resolve.

## Install

```bash
pip install -r categorizer/requirements.txt   # pydantic, pyyaml, langchain-openai, python-dotenv
echo 'OPENAI_API_KEY=sk-...' >> .env           # only needed for the LLM phase
```

The key is read from a `.env` file in your project (auto-loaded, no `export`
needed). An already-exported `OPENAI_API_KEY` environment variable takes
precedence over `.env`.

## Quick start

```python
from categorizer import Categorizer, Taxonomy, cat, to_dataframe

taxonomy = Taxonomy(categories=[            # Python builder…
    cat("Food & Dining",
        cat("Coffee", keywords=["KOFEYNYA"]),
        cat("Groceries")),
    cat("Healthcare",
        cat("Medications", keywords=["ECZANE"])),
])
# …or: Taxonomy.from_yaml("categories.yaml") / from_json(...) / from_dict({...})

c = Categorizer(taxonomy)
results = c.categorize([
    "coffee - kofeynya 30 dollars",         # resolved by keyword, free
    "dinner at a small italian place",      # resolved by the LLM
])

for r in results:
    print(r.path, r.method, r.rationale)    # ['Food & Dining', 'Coffee'] keyword …
df = to_dataframe(results)                  # flat table: lvl1, lvl2, method, ok, …
```

Async: `await c.acategorize(records)`. Inputs: list of strings, dicts,
`Record` objects, or a pandas DataFrame.

Runnable demo: `python -m categorizer.example`

## The three phases

1. **Patterns** (regex, instant, free) — per-source rules that assign a full
   category path in one hit; a record's `source` field selects the rule set.
2. **Keywords** (substring, instant, free) — per-category trigger words
   assign the full path to that category.
3. **LLM** (semantic, costs money) — only the leftovers. Level-by-level walk
   with structured output, so an out-of-taxonomy answer is impossible.
   Records sharing a `keyword` are called once and share the result
   (`method="cache"`), and the cache persists across calls.

Each phase can be toggled (`use_patterns` / `use_keywords` / `use_llm`).

## Batch mode

`batch_size=N` (default 1) sends **up to N records per LLM call — per tree
level, per group** — not the whole run in one call:

- Only records that reach the LLM phase batch at all: patterns, keywords, and
  the cache resolve their records for free first.
- A group = records at the same position in the tree with the same `source`.
  Each group is chunked into batches of ≤ N; all batches fly concurrently
  under `max_concurrency`.
- The walk stays level-by-level: one call judges a batch against the top-level
  categories, then records regroup by the parent each one chose and batch
  again for that parent's children.

Example: 10 records on a 2-level taxonomy = 20 calls per-record, but ~7
batched — 1 call at level 1, then one smaller call per chosen parent at
level 2.

Reliability: batch items are id-keyed, so a missing, duplicated, or invalid
item in a response automatically re-runs as a single call at that level, and a
failed batch call degrades its whole chunk to singles — nothing is silently
dropped. `allow_no_fit` verdicts work inside batches.

When to use it: batching is the **cost and rate-limit** lever — fewer
requests, and the taxonomy tokens amortized across N records. It is not a
latency lever: for small runs, N parallel single calls finish faster than one
call writing N rationales. Reach for it when record counts vastly outnumber
`max_concurrency` or requests-per-minute limits bind, and keep N modest
(≤ ~10–20): per-record accuracy degrades as batches grow.

## Config files

The taxonomy is a typed Pydantic model; file formats are just loaders —
YAML in, JSON in, dict in, Python builder in, same validated `Taxonomy` out.
Typos, duplicate siblings, and patterns pointing at nonexistent categories
fail fast with clear errors.

`categories.yaml` — the taxonomy
(full example: [`categorizer/examples/categories.yaml`](categorizer/examples/categories.yaml)):

```yaml
categories:
  - name: Food & Dining
    rules:                                   # hard constraints shown to the LLM
      - "Supermarket purchases (Migros, BIM, ...) belong here, not in Retail Purchases."
    children:
      - name: Coffee
        keywords: [KOFEYNYA]                 # substring auto-triggers
        description: ""                      # optional context for the LLM
```

`patterns.yaml` — per-source regex rules
(full example: [`categorizer/examples/bank_patterns.yaml`](categorizer/examples/bank_patterns.yaml)):

```yaml
sources:
  QNB Finansbank Enpara:
    - pattern: "Gelen Transfer"
      path: [Incoming P2P Transfers, Incoming Money]
```

Editor autocomplete / CI validation:

```python
Taxonomy.write_file_schema("categories.schema.json")
Patterns.write_file_schema("patterns.schema.json")
# YAML header:  # yaml-language-server: $schema=categories.schema.json
```

> **Steering tip:** the LLM decides one level at a time and only sees that
> level's options. Put `rules`/`description` at the level where the wrong
> turn happens — a helper on a level-2 child cannot fix a level-1 mistake.

## What you get back

Each record yields a `Result`: `path` (root→leaf names), `method`
(`pattern | keyword | llm | cache`), per-level `rationale`, `ok` flag, and
`error` if unresolved. One bad record never stops the batch.

Records that genuinely don't belong anywhere are **signaled, not crammed** into
the nearest bin: `Result.unplaced` carries a grounded verdict —
`no_fit(off_axis | uncovered | under_bar)` when no category can be truthfully
grounded, or `cant_tell` when the text is too opaque to identify. The rationale
states why. No-fit is an outcome, never a category.

## Knobs

```python
Categorizer(
    taxonomy, patterns,
    model="gpt-5.4-nano",        # any OpenAI chat model
    llm=my_langchain_model,      # or bring any LangChain chat model
    use_patterns=True, use_keywords=True, use_llm=True,
    allow_no_fit=True,           # False → forced choice: every record gets a category
    batch_size=1,                # >1 → judge N records per LLM call (cuts requests and
                                 #   amortizes the taxonomy tokens; accuracy degrades as
                                 #   N grows — keep ≤ ~10-20; misses auto-repair as singles)
    cache=True,
    max_concurrency=32,
    on_progress=lambda p: ...,   # Progress(phase, done, total)
    # extra kwargs go to ChatOpenAI, e.g. reasoning_effort="low", timeout=30
)
```
