Metadata-Version: 2.5
Name: varga
Version: 0.0.8
Summary: what kind of thing a document is, and the fields it holds: cue-scored doctypes and schema-constrained extraction
Project-URL: Repository, https://github.com/vedicreader/varga
Project-URL: Documentation, https://vedicreader.github.io/varga/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: doctype,document classification,information extraction,invoice,nbdev,schema,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Requires-Dist: rahasya>=0.0.1
Description-Content-Type: text/markdown

# varga


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

varga (वर्ग, “class”) answers two questions about a document.

The first question is what kind of document it is. The answer is one of 31 kinds, such as an invoice, a contract, a research paper or a bank statement. [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) answers it by counting evidence in the text, such as cue phrases, amounts of money, dates and reference numbers. It uses no model and no network. It also says whether the evidence is clear enough to trust.

The second question is what the document says, in a fixed set of fields. For an invoice, the fields are the number, the date, the vendor and the total. The set of fields is a schema. [`as_schema`](https://vedicreader.github.io/varga/schema.html#as_schema) returns a built-in schema, or a schema that you write as a short string. [`structured`](https://vedicreader.github.io/varga/schema.html#structured) asks a language model that you supply to fill in the schema. [`check_fields`](https://vedicreader.github.io/varga/schema.html#check_fields) then checks the dates and amounts that the model returned.

## Install

``` sh
uv add varga            # document types, with no model and no network
uv add varga rishi      # and a chat model for `structured` to ask
```

varga depends only on fastcore and [rahasya](https://github.com/vedicreader/rahasya). [`structured`](https://vedicreader.github.io/varga/schema.html#structured) needs a chat object, such as a `rishi.Chat`. varga does not import rishi. Any object that has a `.structured` method and can be called to return a reply works.

## What kind of document is it

``` python
INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
Total due: 1,240.00 EUR
Payment terms: net 30
'''
r = guess_type(INVOICE)
r.doctype, r.decisive, round(r.margin, 2)
```

    ('invoice', True, 0.4)

`doctype` is the best guess. `margin` is the score of the best guess minus the score of the runner-up. `decisive` is True when the best score is at least 0.4 and the margin is at least 0.12. When two kinds tie, `margin` is 0 and `decisive` is False. The evidence in the text has then not settled the question.

[`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) never calls a model. `TYPE_SP` in `varga.core` is a system prompt that asks a model to choose one label. You can use it when `decisive` is False.

The evidence comes from [`signals`](https://vedicreader.github.io/varga/core.html#signals), which counts what the text contains:

``` python
signals(INVOICE).counts
```

    {'money': 1, 'date': 1, 'ref': 1, 'heading': 1, 'org': 2}

## What is in it

Each built-in schema is a Python dataclass, so its fields have names and types:

``` python
from dataclasses import fields
[f.name for f in fields(as_schema('invoice'))]
```

    ['number',
     'date',
     'due_date',
     'vendor',
     'vendor_tax_id',
     'bill_to',
     'ship_to',
     'currency',
     'subtotal',
     'tax',
     'total',
     'payment_terms',
     'items']

There are fifteen built-in schema names: `invoice`, `purchase_order`, `quote`, `receipt`, `catalogue`, `contract`, `resume`, `paper`, `meeting_notes`, `id_card`, `passport`, `bank_statement`, `payslip`, `tax_return` and `other`. Some names share a schema. `purchase_order` and `quote` use the invoice schema. You can also write a schema as `'name:type, name:type'`:

``` python
[(f.name, f.type.__name__) for f in fields(as_schema('vendor:str, total:float, items:list'))]
```

    [('vendor', 'str'), ('total', 'float'), ('items', 'list')]

`structured(chat, prompt, schema)` asks the model to fill the schema and returns a dictionary. It first uses the constrained mode of the model, which forces the reply into the shape of the schema. If that call raises, [`structured`](https://vedicreader.github.io/varga/schema.html#structured) warns and asks again for a plain JSON reply.

With `cite=True`, [`structured`](https://vedicreader.github.io/varga/schema.html#structured) returns `{field: {value, quote}}`. The quote is the text that the model says it read the value from. `locate(fields, pages)` finds the page and block of each quote by string matching, with no model. You can then check each value against the source.

## The document types

`DOCTYPES` holds all 31 kinds and the evidence for each:

``` python
len(DOCTYPES), sorted(DOCTYPES)
```

Eight kinds describe work documents: `proposal`, `presentation`, `requirements_spec`, `technical_design`, `regulatory_guidance`, `procedure`, `qa_artifact` and `roadmap`. Two more kinds, `claim` and `clinical_record`, have a similar shape. The tests check that [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) decides each of these ten kinds correctly from its own text without a model. The tests also check that ordinary prose with the same vocabulary scores under 0.2 against all ten.

Six kinds are records about one person: `id_card`, `passport`, `bank_statement`, `payslip`, `tax_return` and `clinical_record`. `SENSITIVE` lists them. The result of [`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type) has `sensitive=True` when its answer is one of them. varga takes no action on this flag. Your program can use it, for example, to keep such a document on the machine and not send it to a hosted model through [`structured`](https://vedicreader.github.io/varga/schema.html#structured).

## Where text goes

[`guess_type`](https://vedicreader.github.io/varga/core.html#guess_type), [`signals`](https://vedicreader.github.io/varga/core.html#signals) and [`check_fields`](https://vedicreader.github.io/varga/schema.html#check_fields) run entirely in your own process and send nothing anywhere. [`structured`](https://vedicreader.github.io/varga/schema.html#structured) sends its `prompt` to the chat object that you pass. The prompt normally contains the document. The text goes to the place where that model runs. varga does not store documents or results.

The name finder in [`signals`](https://vedicreader.github.io/varga/core.html#signals) uses the honorific pattern of rahasya. Both packages use one definition of a name.

## Development

The notebooks in `nbs/` are the source, and the Python modules are generated from them.

``` sh
pip install -e .
nbdev-prepare
```
