Metadata-Version: 2.5
Name: varga
Version: 0.0.7
Summary: what kind of thing a document is, and the fields it holds: cue-scored doctypes and schema-constrained extraction
Project-URL: Repository, https://github.com/vedicreader/varga
Project-URL: Documentation, https://vedicreader.github.io/varga/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: doctype,document classification,information extraction,invoice,nbdev,schema,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Requires-Dist: rahasya>=0.0.1
Description-Content-Type: text/markdown

# varga


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

varga (वर्ग, “class”) answers two questions about a document. The first is what kind of document it is: an invoice, a contract, a research paper, a bank statement, or one of 31 kinds in all. [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) answers it by counting evidence in the text, such as cue phrases, amounts of money, dates and reference numbers. It uses no model and no network, and it says whether the evidence was clear enough to trust.

The second question is what the document says, in a fixed set of fields: an invoice’s number, date, vendor and total, for instance. [`as_schema`](https://vedicreader.github.io/varga/schema.html#as_schema) gives you the set of fields, called a schema, either one of the built-in ones or one you write as a short string. [`structured`](https://vedicreader.github.io/varga/schema.html#structured) asks a language model you supply to fill it in, and [`check_fields`](https://vedicreader.github.io/varga/schema.html#check_fields) then checks the dates and amounts the model returned.

## Install

``` sh
uv add varga            # document types, with no model and no network
uv add varga rishi      # and a chat model for `structured` to ask
```

varga depends only on fastcore and [rahasya](https://github.com/vedicreader/rahasya). [`structured`](https://vedicreader.github.io/varga/schema.html#structured) needs a chat object, such as a `rishi.Chat`, but varga imports nothing to use it: any object with a `.structured` method and a call that returns its reply will do.

## What kind of document is it

``` python
INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
Total due: 1,240.00 EUR
Payment terms: net 30
'''
r = guess_type(INVOICE)
r.doctype, r.decisive, round(r.margin, 2)
```

    ('invoice', True, 0.4)

`doctype` is the best guess and `margin` is how far its score is ahead of the runner-up. `decisive` says whether to trust the guess: it is True when the best score is at least 0.4 and at least 0.12 ahead of the next. When two kinds tie, `margin` is 0 and `decisive` is False, which tells you the text alone has not settled the question. [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) never calls a model itself. If you want a second opinion, `TYPE_SP` in `varga.core` is a ready-made system prompt for asking a model to choose one label.

The evidence comes from [`signals`](https://vedicreader.github.io/varga/core.html#signals), which counts what the text contains:

``` python
signals(INVOICE).counts
```

    {'money': 1, 'date': 1, 'ref': 1, 'heading': 1, 'org': 2}

## What is in it

Each built-in schema is a Python dataclass, so its fields have names and types:

``` python
from dataclasses import fields
[f.name for f in fields(as_schema('invoice'))]
```

    ['number',
     'date',
     'due_date',
     'vendor',
     'vendor_tax_id',
     'bill_to',
     'ship_to',
     'currency',
     'subtotal',
     'tax',
     'total',
     'payment_terms',
     'items']

There are fifteen built-in schema names: `invoice`, `purchase_order`, `quote`, `receipt`, `catalogue`, `contract`, `resume`, `paper`, `meeting_notes`, `id_card`, `passport`, `bank_statement`, `payslip`, `tax_return` and `other`. Some share a schema: purchase orders and quotes use the invoice schema. You can also write a schema on the spot as `'name:type, name:type'`:

``` python
[(f.name, f.type.__name__) for f in fields(as_schema('vendor:str, total:float, items:list'))]
```

    [('vendor', 'str'), ('total', 'float'), ('items', 'list')]

`structured(chat, prompt, schema)` asks the model to fill the schema and returns a dictionary. It first uses the model’s constrained mode, which forces the reply into the schema’s shape. If that call raises, for a model without tool calling for instance, it warns and asks again for a plain JSON reply.

With `cite=True` it returns `{field: {value, quote}}`, where the quote is the text the model says it read the value from. `locate(fields, pages)` then finds the page and block holding each quote by plain string matching, with no model, so that each value can be checked against the source.

## The document types

`DOCTYPES` holds all 31 kinds and the evidence for each:

``` python
len(DOCTYPES), sorted(DOCTYPES)
```

Ten of them describe work documents: `proposal`, `presentation`, `requirements_spec`, `technical_design`, `regulatory_guidance`, `procedure`, `qa_artifact`, `roadmap`, `claim` and `clinical_record`. The tests check that each is decided correctly from its own text without a model, and that ordinary prose which happens to use their vocabulary scores under 0.2 against all of them.

Six kinds are records about one person: `id_card`, `passport`, `bank_statement`, `payslip`, `tax_return` and `clinical_record`. They are listed in `SENSITIVE`, and the result of [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) has `sensitive=True` when its answer is one of them. varga does not act on this itself. It is there so that your program can, for example, keep such a document on the machine instead of sending it to a hosted model through [`structured`](https://vedicreader.github.io/varga/schema.html#structured).

## Where text goes

[`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type), [`signals`](https://vedicreader.github.io/varga/core.html#signals) and [`check_fields`](https://vedicreader.github.io/varga/schema.html#check_fields) run entirely in your own process and send nothing anywhere. [`structured`](https://vedicreader.github.io/varga/schema.html#structured) sends its `prompt`, which normally contains the document, to the chat object you pass it, so the text goes wherever that model runs. varga does not store documents or results.

The name finder in [`signals`](https://vedicreader.github.io/varga/core.html#signals) uses rahasya’s honorific pattern, so the two packages agree on what counts as a name.

## Development

The notebooks in `nbs/` are the source, and the Python modules are generated from them.

``` sh
pip install -e .
nbdev-prepare
```
