Metadata-Version: 2.5
Name: vishalakshi
Version: 0.1.35
Summary: one vault for everything you read: web, papers, video, files and code, searchable together and answerable by a local or hosted model
Project-URL: Repository, https://github.com/vedicreader/vishalakshi
Project-URL: Documentation, https://vedicreader.github.io/vishalakshi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document classification,information extraction,knowledge base,memory,nbdev,ner,rag,research,semantic search,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Requires-Dist: fossick>=0.1.18
Requires-Dist: ganapati>=0.0.1
Requires-Dist: koshas>=0.1.11
Requires-Dist: litesearch>=0.1.61
Requires-Dist: mcp
Requires-Dist: numpy>=1.26
Requires-Dist: pdflite>=0.0.4
Requires-Dist: pobblebonk>=0.0.2
Requires-Dist: pyskills>=0.0.27
Requires-Dist: rahasya>=0.0.5
Requires-Dist: rgapi
Requires-Dist: rishi>=0.1.44
Requires-Dist: uraiyadal>=0.0.13
Requires-Dist: varga>=0.0.7
Requires-Dist: vruksha>=0.1.1
Provides-Extra: rerank
Requires-Dist: flashrank>=0.2.10; extra == 'rerank'
Description-Content-Type: text/markdown

# vishalakshi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

vishalakshi keeps what you collect in one SQLite file on your own machine. The file is called a vault. You search the vault and ask it questions. One question can draw on a paper, a web page, a talk transcript, your own notes and source code on disk. The answer cites the sections it used.

vishalakshi is built from smaller packages. [fossick](https://github.com/vedicreader/fossick) fetches pages, papers and videos. [litesearch](https://github.com/Karthik777/litesearch) stores the text and finds it again by keyword and by meaning. A [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault) is a litesearch [`Index`](https://Karthik777.github.io/litesearch/api.html#index), and its search defaults come from litesearch. [Urai](https://github.com/vedicreader/urai) is the chat interface the vault uses to talk to models. [Rishi](https://github.com/vedicreader/rishi) supplies the model backends. [kosha](https://github.com/vedicreader/kosha) indexes source code. [rgapi](https://github.com/AnswerDotAI/rgapi) searches files on disk that no index covers yet. [rahasya](https://vedicreader.github.io/rahasya/) finds personal information. [varga](https://vedicreader.github.io/varga/) recognises the kind of a document.

## Install

``` sh
pip install vishalakshi
```

## Adding and searching

[`Vault()`](https://vedicreader.github.io/vishalakshi/core.html#vault) with no argument opens `~/.vishalakshi/vault.db`. This page uses a throwaway file and fills it with this repository’s own docs and source.

``` python
from tempfile import mkdtemp
from litesearch import repo_root
from vishalakshi import Vault

root = str(repo_root() or Path('..'))   # works from nbs/ or the repo root
v = Vault(Path(mkdtemp())/'vault.db')
v.enc.note
```

    'minishlab/potion-multilingual-128M (256d, float16, model2vec)'

`add` takes a directory, a file or a piece of text. `grab` takes almost anything else, such as an arXiv id, a YouTube link, a GitHub repository, a PDF or a web address. It reads each kind in the way that suits it ([acquire](https://vedicreader.github.io/vishalakshi/acquire.html)). Adding a source that is already in the vault reads it again only when its content changed. The `changed` field of the result says whether it did. `note` writes your own note. A note with a `key=` or a title replaces the earlier note with the same key or title. A note with neither is added as a new note.

Every search result also has the fields `age` and `stale`. `age` is the number of seconds since the document was filed. `stale` is True when the vault believes the document no longer matches its source. Stale results sort last. `poll()` (below) marks a document stale in three cases: a file changed or vanished, a web page is past its time limit and no watch refreshes it, or a watch has replaced it with a newer copy. `poll()` deletes only documents that are both stale and replaced.

``` python
v.add(root)                       # README.md and every notebook under nbs/
v.note('federate fuses the legs by rank because they share no vector space: the vault embeds '
       'prose, kosha embeds identifiers, ripgrep embeds nothing.', tags=['retrieval', 'design'])
v.note('one document per key: writing it again replaces it', title='federation', key='federation')
v.stats()
```

`ask` needs a language model. The default is `gemma-4-E2B`, a small model that runs on this machine through LiteRT. It uses the GPU when one is present and needs no API key. `model=` names any other model that Rishi knows, local or hosted. `chat_kw=` passes settings to that model. `$VISHALAKSHI_MODEL` changes the default model. `$VISHALAKSHI_GPU=0` keeps LiteRT on the CPU. `ask` retrieves the four best sections (`sections=4`) and shows the model up to 1,500 characters of each (`max_chars=1500`). It returns the answer and [`cited`](https://vedicreader.github.io/vishalakshi/ask.html#cited), the sections that the answer cites ([ask](https://vedicreader.github.io/vishalakshi/ask.html)).

``` python
r = v.ask('why are rankings fused instead of distances?')
print(r.model, '·', r.runtime)
print(r.answer)
print(L(r.cited).map(lambda c: f"{c['n']} {c['breadcrumb']}"))
if (hit := first(r.cited)): print(v.read(hit['node_id'])['text'][:400])
```

    litert/litert-community/gemma-4-E2B-it-litert-lm · litert
    The provided sections do not explicitly state why rankings are fused instead of distances. However, one section mentions that `federate` fuses rankings with RRF and never distances [2].
    ['2 03 code › Fusing legs that share no vector space']
    The vault embeds prose, kosha embeds identifiers, ripgrep embeds nothing. Legs share no vector space, so `federate` fuses rankings with RRF and never distances.


    #| export
    def _prose(v, q, n, kind=None, source:str='prose') -> L:
        'Vault sections, normalised to the federated row shape. `source` names the shelf they came from.'
        return L(v.sections(q, limit=n, kind=kind)).map(
            lambda 

## Personal information and junk

A personal vault holds things that should not leave your machine, such as invoices, letters and bank statements. vishalakshi looks for personal information in everything it stores. It uses patterns and checksums and does not use a model. It finds email addresses, phone numbers, card and account numbers, national identity numbers, street addresses, dates of birth and API keys. On a held-out test set of 240 generated documents it missed nothing and raised no false alarm (`evals/RESULTS.md`). Names are harder. It finds only names that follow a title such as Dr or Mrs, and only when name detection is on (`ner=True`). The `scanned_ner` field of a report says whether name detection ran.

What it finds sets the privacy level of each document. The levels are Open, Personal and Local only. A document is Personal when an identifier was found or someone marked it. A document is Local only when someone set it by hand or when its type is sensitive, such as a passport or a payslip. Each shelf can also have a floor, a level that none of its documents goes below. A hosted model is a model that runs anywhere except this machine. Before every model call the vault decides with `send_check` what each document may give a hosted model. An Open document goes whole. A Personal document goes masked, or does not go. A Local only document never goes. A document’s title, headings and source path follow the same rule as its text. A model on this machine may read everything. The vault records each send to a hosted model in its send log. The details are on [pii](https://vedicreader.github.io/vishalakshi/pii.html).

Junk is a separate problem. Cookie banners and site footers match every search and answer nothing. `suggest_noisy` ranks likely junk without labels (0.988 AUC on the test corpora). `mark_noisy` takes a document out of search ([quality](https://vedicreader.github.io/vishalakshi/quality.html)).

``` python
from vishalakshi.pii import pii_report

r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying, r.kinds
```

    (True,
     {'email': 1, 'card': 1, 'secret': 1},
     {'email': 1, 'card': 1, 'secret': 1})

``` python
# a person's judgement, over the top of the arithmetic
v.add('A letter about Jane, and what she said on Tuesday.',
      title='letter', source='/inbox/letter.md')
v.add('Cookie policy. All rights reserved. Privacy. Terms. Contact us.',
      title='footer', source='/inbox/footer.md')
v.mark_pii('/inbox/letter.md', reason='names a person')     # private even though nothing matched
v.mark_noisy('/inbox/footer.md', reason='site furniture')   # out of search, sections, context, ask

r = v.pii('/inbox/letter.md', ner=True)   # scanned_ner says whether names were looked for
r.scanned_ner, r.detected, r.has_pii, any('footer' in h['breadcrumb'] for h in v.search('cookie policy'))
```

| `pii=` in `ask` | what happens when the retrieved sections hold personal information |
|----|----|
| `local` | a model on this machine answers and is told never to repeat a personal detail. If the model is not local, nothing is sent |
| `redact` | each recognised identifier is masked, then the named model answers, hosted or not. The heading, title and source of each section are masked with its text. Names without a title are not masked |
| `refuse` | no model is asked. The result says what was held back |

The default is `local` when a local runtime is installed, and `redact` when not. `ask` cannot run with detection off. `pii='off'` gives a warning and `ask` applies the default. The `policy` field of the result says which policy was used. The answer has any identifier that the model repeats masked. Under every policy a hosted model receives a Local only section as a placeholder only, with no heading, title or source. Search and reading also take `pii=` (`off`, `redact` or `refuse`). There the default is `off`, which returns the stored text. The MCP server defaults to `redact` for every tool and refuses `off`, because its client is a model that the vault does not run.

``` python
v.mark_not_pii(doc_id, reason='my own invoice')          # a person clears a false alarm
v.set_level(doc_id, 'local')                              # never send this one to a hosted model
v.send_check(refs=[doc_id], model='gpt-4.1-nano').why     # what a send would do, in one sentence
v.accept_noisy(k=10, reason='reviewed')                   # mark the current top junk suggestions
v.fit_noise(save=True); v.use_noise(True)                 # optional: fit the junk score to your marks
```

## Paperwork

The `kind` of a document says how it arrived. Its doctype says what it is, such as an invoice, a receipt, a contract, a paper or a payslip. `categorize` decides the doctype from cue phrases and figures in the text. It asks a model only when the cues cannot decide. `extract` pulls fields out of a document, such as the number, vendor and total of an invoice. Without a `schema`, the doctype picks one. `ask_doc` and `ask(ref=...)` answer a question about named documents. They read the named documents whole and use the rest of the vault as additional context. More on [extract](https://vedicreader.github.io/vishalakshi/extract.html) and [ask](https://vedicreader.github.io/vishalakshi/ask.html).

``` python
INVOICE = '''# INVOICE
Invoice No: ACM-2024-0117
Date: 2024-03-01
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd
Total due: 1,240.00 EUR
Due date: 2024-03-31
Items: widgets x10 @ 124.00
'''
v.add(INVOICE, title='Acme invoice 0117', source='/inbox/acme-0117.md')
v.categorize_all(llm='never')
v.doctypes()
```

    {'code': 13, 'documentation': 4, 'invoice': 1, 'paper': 1, 'other': 1}

``` python
from dataclasses import fields
from vishalakshi.extract import SCHEMAS, as_schema

print(L(SCHEMAS), L(fields(as_schema('invoice'))).attrgot('name'))
e = v.extract('/inbox/acme-0117.md')                    # doctype picks schema
a = v.ask_doc('/inbox/acme-0117.md', 'what is owed, to whom, and by when?',
              schema='amount:float, payee:str, due:str')
e.schema, e.fields, a
```

    [{'invoice': <class 'vishalakshi.extract.Invoice'>, 'purchase_order': <class 'vishalakshi.extract.Invoice'>, 'quote': <class 'vishalakshi.extract.Invoice'>, 'receipt': <class 'vishalakshi.extract.Receipt'>, 'catalogue': <class 'vishalakshi.extract.Catalogue'>, 'contract': <class 'vishalakshi.extract.Contract'>, 'resume': <class 'vishalakshi.extract.Resume'>, 'paper': <class 'vishalakshi.extract.Paper'>, 'meeting_notes': <class 'vishalakshi.extract.MeetingNotes'>, 'other': <class 'vishalakshi.extract.Summary'>}] ['number', 'date', 'due_date', 'vendor', 'vendor_tax_id', 'bill_to', 'ship_to', 'currency', 'subtotal', 'tax', 'total', 'payment_terms', 'items']

    /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/extract.py:580: UserWarning: ValueError on a constrained call for Answer (model neither called the tool nor returned JSON; reply: 'The invoice details are as follows:\n*   **Invoice No:** ACM-2024-0117 [1]\n*   **Date:** 202); retrying as a JSON reply.
      warnings.warn(f'{type(e).__name__} on a constrained call for {schema.__name__} '

    ('Invoice',
     {'number': 'ACM-2024-0117',
      'date': '2024-03-01',
      'due_date': '',
      'vendor': 'Acme Supplies Ltd',
      'vendor_tax_id': '',
      'bill_to': 'Contoso GmbH, Berlin',
      'ship_to': '',
      'currency': 'EUR',
      'subtotal': 1240.0,
      'tax': 0,
      'total': 1240.0,
      'payment_terms': '',
      'items': ['widgets x10 @ 124.00']},
     {'question': 'what is owed, to whom, and by when?',
      'model': 'litert/litert-community/gemma-4-E2B-it-litert-lm',
      'runtime': 'litert',
      'context': {'results': [{'node_id': '4c1bdc4b329e3c3d#0', 'title': 'Acme invoice 0117', 'doc_id': '4c1bdc4b329e3c3d', 'breadcrumb': 'Acme invoice 0117', 'filename': '/inbox/acme-0117.md', 'pages': None, 'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'}, {'node_id': 'e8024c5880508178#0', 'title': '05 mcp', 'doc_id': 'e8024c5880508178', 'breadcrumb': '05 mcp', 'filename': None, 'pages': (0, 0), 'text': "    'files anything you point it at; `note` writes your own conclusions back so they are searched '\n    'alongside the sources. `add_tree` points at a directory and splits it: documents into the '\n    'vault, source files into kosha. When kosha has indexed '\n    'the repo, `context` and `ask` add code sections to what they retrieve on their own, so a '\n    'question about the user\\'s own system is\n\n    'For one document rather than the corpus: `document` returns the whole of it, `categorize` says '\n    'what kind of thing it is (invoice, catalogue, contract, paper, …), `extract` pulls its fields '\n    'out against a schema (a name like `invoice`, or a spec like `vendor:str, total:float` you '\n    'make up on the spot), and `ask_doc` answers a question about that document with the rest of '\n "}, {'node_id': 'bd083e908521c339#24', 'title': '`chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at', 'doc_id': 'bd083e908521c339', 'breadcrumb': '06 extract › `chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at', 'filename': None, 'pages': (0, 0), 'text': "test_eq(a.cited.attrgot('node_id')[0], f'{a.doc_id}#0')\nassert 'VAT (20%)' in a.prompt, 'the whole document must reach the model, not a retrieved chunk'\nassert a.prompt.endswith('Question: what is the total due?')\n# the same question as data instead of prose, through a schema built at the moment of asking\nd = _x.ask_doc('/inbox/acme-0117.md', 'what is owed and to whom?',\n               schema='amo\n\ntest_eq(_x.document(_p).origin, 'disk')\ntest_eq(_x.categorize(str(_p), llm='never', save=False).doctype, 'meeting_notes')\n\n#| eval:false\n#| hide\n# the route that had to wait: what acquisition could not tell at the door, the doctype can\nv.shelf('papers', offline=True)          # pre-registered, so this test pays for no encoder download\nr = v.reshelf('Acme invoice', llm='never')\ntest_eq((r.doctype, \n\nanswer), ('Answer', ['amount', 'currency', 'owed_to'], None))\ntest_eq(d.fields['amount'], 180.0)\n# and it is `ask` underneath\ntest_eq(_x.ask('what is the total due?', ref='/inbox/acme-0117.md', related=2, **_m).answer, a.answer)\n\n# a file the vault has never seen, straight off disk: no model needed to prove where it came from\nfrom fastcore.all import Path\nfrom tempfile import mkdtemp\n_p = Path(mkd"}, {'node_id': '10bdb7299a36ecf7#0', 'title': '02 ask', 'doc_id': '10bdb7299a36ecf7', 'breadcrumb': '02 ask', 'filename': None, 'pages': (0, 0), 'text': 'transcripts, files and their own notes. Answer only from those sections.\n\nRules:\n- Cite every claim with the bracketed number of the section it came from, like [2]. A sentence\n  drawing on two sections cites both.\n- If the sections do not answer the question, say exactly what is missing rather than filling the\n  gap from memory. A vault that admits a hole is useful; one that guesses is not.\n- Sect\n\n- Say what you are holding back and why, in one line, so the questioner knows something is \\\nthere rather than assuming there is nothing.\n- If the question cannot be answered without a personal detail, do not answer it. Say what \\\ninstruction would let you answer it usefully (a count, a comparison, a yes or no, a total) \\\nand stop. The questioner will send that instruction back and you will get an\n\nwhich must never be shown any of it.\n\nRules, in order of importance:\n- Never reproduce a personal detail. No names, addresses, emails, phone numbers, account or \\\ncard numbers, dates of birth, or medical specifics: not in your answer, not as an example, \\\nnot to show your working, not even partially or obfuscated.\n- Answer at the level of shape and quantity instead: how many, what kind, which peri'}],
       'related': [],
       'encoder': 'minishlab/potion-multilingual-128M (256d, float16, model2vec)',
       'doc': {'doc_id': '4c1bdc4b329e3c3d',
        'title': 'Acme invoice 0117',
        'source': '/inbox/acme-0117.md',
        'kind': 'file',
        'meta': {'doctype': 'invoice',
         'doctype_by': 'cues (ner+regex)',
         'doctype_score': 0.7},
        'pages': 1,
        'origin': 'vault',
        'nodes': 2,
        'chars': 179,
        'truncated': False,
        'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'},
       'docs': [{'doc_id': '4c1bdc4b329e3c3d', 'title': 'Acme invoice 0117', 'source': '/inbox/acme-0117.md', 'kind': 'file', 'meta': {'doctype': 'invoice', 'doctype_by': 'cues (ner+regex)', 'doctype_score': 0.7}, 'pages': 1, 'origin': 'vault', 'nodes': 2, 'chars': 179, 'truncated': False, 'text': '# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00'}],
       'n_docs': 1,
       'note': '[1] is the document being asked about; the rest is context from elsewhere in the vault.\n\n'},
      'encoder': 'minishlab/potion-multilingual-128M (256d, float16, model2vec)',
      'prompt': "[1] Acme invoice 0117\n(source: /inbox/acme-0117.md)\n\n# INVOICE\nInvoice No: ACM-2024-0117\nDate: 2024-03-01\nBill to: Contoso GmbH, Berlin\nFrom: Acme Supplies Ltd\nTotal due: 1,240.00 EUR\nDue date: 2024-03-31\nItems: widgets x10 @ 124.00\n\n---\n\n[2] 05 mcp\n(source: e8024c5880508178, pages 0–0)\n\n    'files anything you point it at; `note` writes your own conclusions back so they are searched '\n    'alongside the sources. `add_tree` points at a directory and splits it: documents into the '\n    'vault, source files into kosha. When kosha has indexed '\n    'the repo, `context` and `ask` add code sections to what they retrieve on their own, so a '\n    'question about the user\\'s own system is\n\n    'For one document rather than the corpus: `document` returns the whole of it, `categorize` says '\n    'what kind of thing it is (invoice, catalogue, contract, paper, …), `extract` pulls its fields '\n    'out against a schema (a name like `invoice`, or a spec like `vendor:str, total:float` you '\n    'make up on the spot), and `ask_doc` answers a question about that document with the rest of '\n \n\n---\n\n[3] 06 extract › `chat_kw` reaches the rest of the constructor: `CachedChat` takes `path`, and pointing it at\n(source: bd083e908521c339, pages 0–0)\n\ntest_eq(a.cited.attrgot('node_id')[0], f'{a.doc_id}#0')\nassert 'VAT (20%)' in a.prompt, 'the whole document must reach the model, not a retrieved chunk'\nassert a.prompt.endswith('Question: what is the total due?')\n# the same question as data instead of prose, through a schema built at the moment of asking\nd = _x.ask_doc('/inbox/acme-0117.md', 'what is owed and to whom?',\n               schema='amo\n\ntest_eq(_x.document(_p).origin, 'disk')\ntest_eq(_x.categorize(str(_p), llm='never', save=False).doctype, 'meeting_notes')\n\n#| eval:false\n#| hide\n# the route that had to wait: what acquisition could not tell at the door, the doctype can\nv.shelf('papers', offline=True)          # pre-registered, so this test pays for no encoder download\nr = v.reshelf('Acme invoice', llm='never')\ntest_eq((r.doctype, \n\nanswer), ('Answer', ['amount', 'currency', 'owed_to'], None))\ntest_eq(d.fields['amount'], 180.0)\n# and it is `ask` underneath\ntest_eq(_x.ask('what is the total due?', ref='/inbox/acme-0117.md', related=2, **_m).answer, a.answer)\n\n# a file the vault has never seen, straight off disk: no model needed to prove where it came from\nfrom fastcore.all import Path\nfrom tempfile import mkdtemp\n_p = Path(mkd\n\n---\n\n[4] 02 ask\n(source: 10bdb7299a36ecf7, pages 0–0)\n\ntranscripts, files and their own notes. Answer only from those sections.\n\nRules:\n- Cite every claim with the bracketed number of the section it came from, like [2]. A sentence\n  drawing on two sections cites both.\n- If the sections do not answer the question, say exactly what is missing rather than filling the\n  gap from memory. A vault that admits a hole is useful; one that guesses is not.\n- Sect\n\n- Say what you are holding back and why, in one line, so the questioner knows something is \\\nthere rather than assuming there is nothing.\n- If the question cannot be answered without a personal detail, do not answer it. Say what \\\ninstruction would let you answer it usefully (a count, a comparison, a yes or no, a total) \\\nand stop. The questioner will send that instruction back and you will get an\n\nwhich must never be shown any of it.\n\nRules, in order of importance:\n- Never reproduce a personal detail. No names, addresses, emails, phone numbers, account or \\\ncard numbers, dates of birth, or medical specifics: not in your answer, not as an example, \\\nnot to show your working, not even partially or obfuscated.\n- Answer at the level of shape and quantity instead: how many, what kind, which peri\n\n---\n\n[1] is the document being asked about; the rest is context from elsewhere in the vault.\n\nQuestion: what is owed, to whom, and by when?",
      'answer': None,
      'thinking': '',
      'cited': [],
      'schema': 'Answer',
      'fields': {'amount': 1240.0,
       'payee': 'Contoso GmbH, Berlin',
       'due': '2024-03-31'},
      'doc_id': '4c1bdc4b329e3c3d',
      'title': 'Acme invoice 0117',
      'source': '/inbox/acme-0117.md',
      'origin': 'vault',
      'chars': 179,
      'truncated': False,
      'pii': {'has_pii': False,
       'kinds': {},
       'identifying': {},
       'n': 0,
       'scanned': 3395,
       'density': 0.0,
       'spans': []},
      'usage': total=1,674|in=1,623|out=51|turns=1})

## Code

`index_code` builds a [kosha](https://vedicreader.github.io/vishalakshi/code.html) index of a repository. The index holds the functions and classes, their names and the call graph. When the index exists, `context` and `ask` add code sections next to the prose sections (`code=`). A code hit has no `node_id`. It is cited by `path:line` on disk. `grep` searches the files on disk with ripgrep and needs no index. `federate` merges prose, code and ripgrep results into one ranking.

``` python
v.index_code(root)                # fills .kosha/; context then appends code sections
c = v.context('where do topic nodes get rebuilt?', sections=3, related=0, code=3, dir=root)
# code hits have no node_id; their handle is path:line on disk
c.code, L(c.results).filter(lambda r: r.node_id is None).attrgot('breadcrumb')
```

    parse files from /Users/71293/code/personal/orgs/vishalakshi: 100%|██████████| 18/18 [00:00<00:00, 225.97it/s]

<style>
    progress { appearance: none; border: none; border-radius: 4px; width: 300px;
        height: 20px; vertical-align: middle; background: #e0e0e0; }
&#10;    progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; }
    progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; }
    progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; }
&#10;    progress:not([value]) {
        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); }
&#10;    progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted { background: #F44336; }    
&#10;    table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; }
    table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; }
    table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; }
    table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; }
</style>

<style>
    progress { appearance: none; border: none; border-radius: 4px; width: 300px;
        height: 20px; vertical-align: middle; background: #e0e0e0; }
&#10;    progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; }
    progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; }
    progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; }
&#10;    progress:not([value]) {
        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); }
&#10;    progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted { background: #F44336; }    
&#10;    table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; }
    table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; }
    table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; }
    table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; }
</style>

    (3,
     ['repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/core.py:520', 'grep › README.md:119', 'repo › /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/core.py:547'])

``` python
L(v.grep('rrf_all', root, limit=4)).attrgot('where')   # ripgrep; no kosha needed
```

    ['README.md:125', 'nbs/index.ipynb:573', 'vishalakshi/code.py:12', 'vishalakshi/code.py:128']

## Watches and the queue

A watch repeats an acquisition on a schedule. Its action is an acquisition method (`url`, `web`, `harvest`, `arxiv`, `youtube`, `crawl` or `path`), `remind` to write a note, or `folder`. `kind=` is another name for `action=`. A `folder` watch keeps a snapshot of a directory. When it fires, it reports what was added, changed or removed, with its `instructions` for the person who reviews the change. A schedule is `every='6h'` or `cron='0 9 * * 1'`. [pobblebonk](https://vedicreader.github.io/pobblebonk/) keeps the schedules in the vault file.

`poll()` is the tick to call from cron or from the timer of an app. It returns jobs whose worker died to the queue, queues the watches that are due, runs the queue, and runs `housekeep` every ten minutes. A failed fetch is retried with a growing delay. After five attempts the job is set aside as dead. `jobs(state='dead')` lists dead jobs. See [jobs](https://vedicreader.github.io/vishalakshi/jobs.html).

``` python
v.watch('https://example.com/changelog', action='url', every='6h')
v.watch('late chunking retrieval', action='web', every='1d', n=5)
v.watch('Re-read the evals', action='remind', every='1w')
v.watch(Path(root)/'nbs', kind='folder', every='1h', instructions='flag any print() left in a notebook', pattern='*.ipynb')
L(v.watches()).map(lambda w: (w['kind'], w['target'][:34], w['every'], w['pattern']))
```

## Books and scans

A scanned book has no usable text layer. An old print set in a legacy font can have a text layer that reads as gibberish. [`read_doc`](https://vedicreader.github.io/vishalakshi/read.html#read_doc) reads a PDF or a page image page by page. With `parse='auto'` it picks a route for each page. It keeps a text layer that passes its checks. It runs OCR on a scanned page. It sends a page that is still scanned, or that is flagged as garbled, to a vision model, local or hosted. `preset=` picks the reading prompt (`generic`, `sanskrit`, `tamil` or `tables`). `hint=` adds a sentence about the book. Given a folder, [`read_doc`](https://vedicreader.github.io/vishalakshi/read.html#read_doc) reads every PDF and image under it. It keeps the text, route and flags of each page as numbered versions. Reading a page again with a better model (`reparse`) never loses the earlier reading. A page image cannot be masked. A Personal page goes to a hosted model only with `allow=True`. A Local only page never goes. [`CachedChat`](https://vedicreader.github.io/vishalakshi/ask.html#cachedchat) records readings and replays them. A page with no recording reaches a model only when `$VISHALAKSHI_RECORD_CHAT` is set or a `chat=` is passed ([read](https://vedicreader.github.io/vishalakshi/read.html)).

`proofread` has other models read the pages again. It files the word changes that each reading would make. A change that two readers agree on is `proposed`. A change that only one reader makes is in doubt. Nothing changes until the owner decides. `accept` applies changes as a new version of the page. `reject` sets changes aside. `undo_decision` and `revert_page` reverse decisions ([proof](https://vedicreader.github.io/vishalakshi/proof.html)).

``` python
v.read_doc('books/', parse='auto', model='claude/sonnet', preset='sanskrit', hint='verses end in ॥ n ॥')
doc = v.read_doc('scan.pdf', parse='llm', model='gemini/gemini-2.5-flash')['doc_id']
v.pages(doc)                                   # text, route (text, ocr, model, proof), flags, version
v.reparse(doc, pages=[3, 4], model='claude/opus')
v.proofread(doc, ['claude/sonnet', 'gemini/gemini-2.5-pro'])
v.changes(doc, 'proposed'); v.accept(doc); print(v.proof_md(doc))
read_doc('scan.pdf', parse='auto', model='claude/sonnet')   # no vault: the pages alone
```

## The rest

| page | what is on it |
|----|----|
| [core](https://vedicreader.github.io/vishalakshi/core.html) | [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault), documents and sections, shelves, marks, `context`, topic maps |
| [acquire](https://vedicreader.github.io/vishalakshi/acquire.html) | `grab`, `url`, `web`, `crawl`, `arxiv`, `pdf`, `youtube`, `github`, `apis`, `harvest`, images, watches, stopping a long ingest with `stop` and `progress` |
| [ask](https://vedicreader.github.io/vishalakshi/ask.html) | `ask`, citations, the privacy policies, `send_check` and the send log, [`CachedChat`](https://vedicreader.github.io/vishalakshi/ask.html#cachedchat) |
| [code](https://vedicreader.github.io/vishalakshi/code.html) | kosha, `symbol`, `where_to_add`, `grep`, `federate` |
| [cli](https://vedicreader.github.io/vishalakshi/cli.html) | vault methods as `vishalakshi` commands |
| [mcp](https://vedicreader.github.io/vishalakshi/mcp.html) | `vishalakshi-mcp`, the vault as MCP tools |
| [extract](https://vedicreader.github.io/vishalakshi/extract.html) | doctypes from [varga](https://vedicreader.github.io/varga/): `categorize`, `extract`, `extract_all`, `ask_doc`, `reshelf` |
| [concepts](https://vedicreader.github.io/vishalakshi/concepts.html) | embedding models, model backends, shelves, and when to change the defaults |
| [skill](https://vedicreader.github.io/vishalakshi/skill.html) | the cheat sheet an agent loads, and what it may call |
| [pii](https://vedicreader.github.io/vishalakshi/pii.html) | personal information over [rahasya](https://vedicreader.github.io/rahasya/): levels, the floor, findings, marks, [`gated`](https://vedicreader.github.io/vishalakshi/pii.html#gated) |
| [quality](https://vedicreader.github.io/vishalakshi/quality.html) | the noise score, `suggest_noisy` and `accept_noisy`, feedback, the ranker |
| [jobs](https://vedicreader.github.io/vishalakshi/jobs.html) | [`Queue`](https://vedicreader.github.io/vishalakshi/jobs.html#queue): retries, leases, dead jobs |
| [read](https://vedicreader.github.io/vishalakshi/read.html) | [`read_doc`](https://vedicreader.github.io/vishalakshi/read.html#read_doc), `reparse`, page versions, presets, caching, progress and cancelling |
| [proof](https://vedicreader.github.io/vishalakshi/proof.html) | `proofread` and the review queue: `changes`, `accept`, `reject`, `edit_change`, `undo_decision`, `revert_page` |

``` sh
vishalakshi grab https://example.com/post
vishalakshi ask "why does late chunking help"
```

| variable | what it sets |
|----|----|
| `$VISHALAKSHI_VAULT` | the vault file |
| `$VISHALAKSHI_MODEL` | the default model |
| `$VISHALAKSHI_PII_MODEL` | the local model that answers when sections are private |
| `$VISHALAKSHI_GPU=0` | keeps LiteRT on the CPU |
| `$VISHALAKSHI_OFFLINE` | never download a model; use the hashing encoder |
| `$VISHALAKSHI_CHAT_CACHE` | the folder [`CachedChat`](https://vedicreader.github.io/vishalakshi/ask.html#cachedchat) records replies in |
| `$VISHALAKSHI_RECORD_CHAT` | lets [`CachedChat`](https://vedicreader.github.io/vishalakshi/ask.html#cachedchat) ask a real model when it has no recording |

The MCP client configuration is on [mcp](https://vedicreader.github.io/vishalakshi/mcp.html). Retrieval trade-offs and the measured defaults are in [concepts](https://vedicreader.github.io/vishalakshi/concepts.html) and [`evals/RESULTS.md`](../evals/RESULTS.md).

## Development

The notebooks in `nbs/` are the source, and `nbdev-export` writes the modules from them.

``` sh
pip install -e .
nbdev-prepare
```
