Metadata-Version: 2.5
Name: vishalakshi
Version: 0.1.4
Summary: one vault for everything you read: web, papers, video, files and code, searchable together and answerable by a local or hosted model
Project-URL: Repository, https://github.com/vedicreader/vishalakshi
Project-URL: Documentation, https://vedicreader.github.io/vishalakshi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document classification,information extraction,knowledge base,memory,nbdev,ner,rag,research,semantic search,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: fastcore>=1.12.31
Requires-Dist: fossick>=0.1.7
Requires-Dist: koshas>=0.0.36
Requires-Dist: litesearch[sanskrit]>=0.1.25
Requires-Dist: mcp
Requires-Dist: numpy>=1.26
Requires-Dist: pyskills>=0.0.27
Requires-Dist: rgapi
Requires-Dist: rishi[litert,remote]>=0.1.2
Provides-Extra: test
Requires-Dist: diskcache; extra == 'test'
Description-Content-Type: text/markdown

# vishalakshi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Install

``` sh
pip install 'vishalakshi[all]'   # + rishi to answer, + kosha & rgapi for code, + mcp for the server
pip install vishalakshi          # vault + acquisition only; no LLM, no MCP
```

## The loop

Everything in this section runs. The corpus is this repository’s own documentation and source, so
the page is reproducible from a clone, and the vault is a throwaway file rather than your real one.
[`Vault()`](https://vedicreader.github.io/vishalakshi/core.html#vault) with no argument uses `~/.vishalakshi/vault.db`.

``` python
from tempfile import mkdtemp
from vishalakshi import Vault

v = Vault(Path(mkdtemp())/'vault.db')
v.enc.note
```

    'minishlab/potion-multilingual-128M (256d, float16, model2vec)'

Ingestion is one call. `add` takes a directory, a file, or text, and routes on which it got.

``` python
v.add('..')                       # README.md and every notebook under nbs/
v.note('federate fuses the legs by rank because they share no vector space: the vault embeds '
       'prose, kosha embeds identifiers, ripgrep embeds nothing.', tags=['retrieval', 'design'])
```

    {'doc_id': '72a08ba6ff19470a',
     'title': 'federate fuses the legs by rank because they share no vector space: the vault em',
     'kind': 'note',
     'nodes': 2,
     'chunks': 1}

Then reach outward. `grab` routes on what the target is: an arXiv id, a YouTube link, a GitHub
repo, a PDF, a local file, a directory. It is the one call a CLI or an agent needs.

``` python
v.grab('1706.03762')                                       # an arXiv id: metadata + abstract
v.pdf('https://arxiv.org/pdf/1706.03762',                  # and the full paper, page by page
      title='Attention Is All You Need')
```

    [2026-08-12 11:16:48] INFO: Fetched (200) <GET https://arxiv.org/pdf/1706.03762> (referer: https://www.google.com/)
    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream

    {'doc_id': '47e4bf9c7b60a727',
     'title': 'Attention Is All You Need',
     'kind': 'pdf',
     'nodes': 5,
     'chunks': 102}

Now ask it something. The model is the package default: `gemma-4-E2B` on LiteRT, on the GPU, no
API key and nothing to configure. Naming a model is naming it to rishi, so an id, a path or a
`runtime/...` prefix all work, and `chat_kw=` reaches the rest of rishi’s constructor. There is no
model registry here to fall out of date. `$VISHALAKSHI_MODEL` replaces the id; `$VISHALAKSHI_GPU=0`
puts LiteRT back on the CPU.

The defaults hand the model four sections at 1500 characters each. That is the point of retrieving
*sections* rather than documents: the answer needs the paragraphs that bear on the question, not the
corpus. A pointed context is why a 2B model on a laptop can answer at all.

``` python
from rishi.litert import gemma4_e2b, gemma4_e4b

r = v.ask('why are rankings fused instead of distances?')
print(r.model, '·', r.runtime)
print(r.answer)
```

    litert/litert-community/gemma-4-E2B-it-litert-lm · litert
    The provided sections indicate that the reason for fusing rankings instead of distances is that the legs share no vector space [2]. This is because the vault embeds prose, kosha embeds identifiers, and ripgrep embeds nothing [2]. Reciprocal Rank Fusion needs only each leg's ordering, which is what survives a change of encoder [2].

Every `[n]` in the answer resolves to a `node_id` you can read. That round trip is the point: a claim you can check against the text it came from.

``` python
for c in r.cited: print(c['n'], c['breadcrumb'])
print()
print(v.read(r.cited[0]['node_id'])['text'][:400])   # the exact text behind the claim
```

    2 03 code › Fusing legs that share no vector space

    The vault embeds prose, kosha embeds identifiers, ripgrep embeds nothing, so the legs cannot be
    merged by distance. Reciprocal Rank Fusion needs only each leg's *ordering* — which is exactly
    what survives a change of encoder — and it is the same mechanism litesearch already uses to
    combine FTS with vectors. Each leg is tried independently: `legs` reports what each contributed,
    or why it did not.

## When the answer would leave the machine

A vault fills up with things that are nobody else’s business: a bank statement, a medical letter, an
exported chat. Answering out of those is what a vault is *for*. Sending them to a hosted API is what
it is not — so `ask` decides, and the decision is arithmetic rather than a model. A classifier that
has to read the document in order to say whether the document may be read has already lost.

``` python
from vishalakshi.pii import pii_report

r = pii_report('Invoice 4471 for Ada Lovelace, ada@example.com. Card 4111 1111 1111 1111. '
               'Order 4111 1111 1111 1112. Server 10.0.0.14 returned 500.')
r.has_pii, r.identifying, r.kinds
```

    (True, {'email': 1, 'card': 1}, {'email': 1, 'card': 1, 'ip': 1})

Two things are doing work there. The card is a card because it passes Luhn, and the order number is
not one because it fails the same check — a checksum is the difference between a detector and a
superstition. And the IP address is *reported* but not *identifying*: a server log is not somebody’s
private life, and treating it as one would cost every question about infrastructure a smaller model
for nothing.

`ask` gates on the sections retrieval actually chose, not on the vault and not on the document, so a
vault holding one bank statement among four hundred papers is only a private question when the
statement is in the room. When it is, `pii='local'` (the default) answers on a local model under a
system prompt that tells it to give shape and quantity instead of detail — and *ignores* the model
you named, because a hosted one cannot be sent the sections:

``` python
v.add('Invoice 4471 for Ada Lovelace, ada@example.com. Card 4111 1111 1111 1111. '
      'Amount 240.00 GBP, due 2026-09-01.', title='invoice 4471', source='/inbox/4471.md')

r = v.ask('what is on invoice 4471?', model='gpt-4.1-nano')   # hosted, and named at the call site
print(r.runtime, '·', r.model)                                # ...and not what answered
print(r.answer)
```

    litert · litert/litert-community/gemma-4-E2B-it-litert-lm
    I am holding back the specific personal details because I cannot reproduce them.
    The document is an invoice.

The check is on the chat object after it is built, so a caller that *lends* a hosted chat — by
mistake, by a stale config — is refused rather than trusted, and nothing is sent. The other three
settings are [`redact`](https://vedicreader.github.io/vishalakshi/pii.html#redact) (mask what arithmetic can recognise, then let a hosted model answer, which
gives up whatever turns on the details and does not mask names), `refuse` (return the finding and no
answer), and `off`. The same arithmetic runs over the answer on the way back out, where a slip costs
a masked token instead of somebody’s account number.

## What is decided, and what is yours

The retrieval defaults are litesearch’s, and they are measured rather than chosen: `evals/` runs 120
known-item queries per genre over three corpora and scores section-level MRR. You inherit the whole
ladder by opening a vault.

| decided for you | why |
|----|----|
| 512-character chunks | +0.06 to +0.12 weighted MRR over page-sized |
| `pre()` on the keyword leg | +0.016 to +0.093 |
| HNSW vector index | -0.005 quality for a large speedup |
| a document tree, always | ranking is a wash, and `toc`/`read`/`sections`/`context` come free |
| one static encoder, 256d, float16 | the spread across four encoders is 0.018 to 0.046, and the static one wins a genre at ~1,700x cheaper indexing |
| Devanagari and IAST fold to the same token | 1.000 verse recall for every encoder tested, from the tokenizer rather than the embedding |

Five things are yours, and none of them is required to get an answer.

| yours | what it costs |
|----|----|
| `rerank=True` on `search`, `sections`, `context` | +0.026 to +0.077 weighted MRR, positive in all twelve paired cells, at roughly 10x the query latency |
| `shelf(name)` | a partition, so two corpora stop diluting each other’s ranking |
| `llm=` on `categorize` and `extract` | the cue table answers most documents for free; a model is for the ties |
| `db.graph_search` | the graph leg by name, for bridge queries |
| `pii=` on `ask` | `local` keeps a private question on a local model, [`redact`](https://vedicreader.github.io/vishalakshi/pii.html#redact) masks the sections and lets a hosted one answer |

The graph leg is off for ranking and stays off. On ordinary known-item questions it costs 0.070 to
0.160 weighted MRR, negative in every genre and flavour measured. It wins only when the answer
shares no word with the question. `connect()` is still worth running, for the reason below.

Sanskrit is the one place something switches itself on. Ingest a Sanskrit file and the vault
re-registers litesearch’s readers with vidyut lemmas and Monier-Williams glosses, because putting
the English behind the Sanskrit into the index beat changing the encoder when it was measured on
this vault’s own shelf: a static encoder with glosses beat a 300M ONNX transformer without them. It
is an ~83 MB download, paid once, only by someone actually reading Sanskrit.

## Everything in one corpus

Every document carries a `kind`: `web`, `pdf`, `arxiv`, `youtube`, `file`, `code`, `data`, `note`,
or whatever litesearch’s parser called the file. Filter when you want to, don’t when you don’t.

``` python
v.stats()
```

    {'docs': 18,
     'nodes': 161,
     'chunks': 817,
     'encoder': 'model2vec',
     'entities': 0,
     'path': '/var/folders/kg/9vdw4mdd1fs58svgh4k1qhr09x7dqh/T/tmpv1o2urwb/vault.db',
     'by_kind': {'notebook': 11,
      'md': 3,
      'txt': 1,
      'pdf': 1,
      'note': 1,
      'file': 1}}

``` python
len(v.search('rank fusion')), len(v.search('rank fusion', kind='note')), len(v.search('rank fusion', kind='md,notebook'))
```

    (10, 1, 10)

The filter is a SQL `WHERE` pushed into the search, not a pass over the results afterwards, so a
narrow filter over a large vault still returns a full page of hits.

Notes are ordinary documents, deliberately. The graph, the clusters and `context()` all see them for
free, so what you concluded about a corpus comes back next to the evidence you concluded it from.

``` python
L(v.sources()).map(lambda d: (d['kind'], d['title'], d['source']))[:6]
```

    [('file', 'invoice 4471', '/inbox/4471.md'), ('pdf', 'Attention Is All You Need', 'https://arxiv.org/pdf/1706.03762'), ('note', 'federate fuses the legs by rank because they share no vector space: the vault em', 'note:c63bff170d37'), ('md', 'CHANGELOG', '../CHANGELOG.md'), ('md', 'README', '../README.md'), ('notebook', '00 core', '../nbs/00_core.ipynb')]

## When the graph and the clusters pay off

`connect()` builds an entity graph over everything in the vault, and `map()` reads the topics it
persisted. Neither is retrieval. The graph leg loses on ordinary queries, which is why it is not
wired into `search`.

What they answer is the question you ask *before* you know what to search for. `map()` is the shape
of what you have collected, which is how you notice that four months of reading turned into three
subjects rather than the one you thought. The graph is for reaching across shelves and across
documents that never share a word: a bridge query where the paper and the invoice both mention a
company neither of them names in the same way. Run `connect()` after a batch of ingests rather than
inside each one, since it rebuilds over the whole vault.
`topic_tree` puts the documents under each label, with the number of chunks each contributes.

That second number is the one worth reading. A topic carried by a single document is usually that
document talking to itself, and a label to distrust. A topic spread across five is a thread actually
running through the corpus, and the place to start reading. It is also how you notice that two
labels you thought were separate subjects are the same source seen twice.

Three queries and no clustering: `connect()` already persisted the topic nodes, so this only pivots
the mentions back onto documents.

``` python
v.connect(n_workers=0)
```

    {'entities': 4513,
     'mentions': 8676,
     'edges': 2264,
     'windows': 2753,
     'resolved': {'merged': 1244,
      'by_ann': 678,
      'by_lexical': 566,
      'edges': 1912,
      'entities': 4513,
      'resolvable': 4513,
      'canonical': 3269},
     'topics': 161,
     'method': 'knn'}

``` python
v.show_topics(limit=5, docs=3)      # v.topic_tree() returns the same thing as data
```

    cues, ents, needs, org                       (8 chunks, 1 docs)
      `- 06 extract                                  8
    blob, github, https, url                     (8 chunks, 3 docs)
      |- 01 acquire                                  6
      |- README                                      1
      `- SKILL                                       1
    schema, as_schema, dataclass, float          (8 chunks, 2 docs)
      |- 06 extract                                  6
      `- 02 ask                                      2
    ast, loading, appends, index_code            (8 chunks, 5 docs)
      |- SKILL                                       3
      |- 01 acquire                                  2
      `- index                                       1
    doctype, save, schema, doc_id                (8 chunks, 1 docs)
      `- 06 extract                                  8

## What each document is, and what is inside it

`kind` says how a document arrived. What it *is* is a different question, and one worth answering:
a vault that knows which of its documents are invoices can hand you their totals as a table. The cue
table knows 26 answers, in two halves — what arrives from outside (an invoice, a price list, a
contract, a paper, a transcript, a source file) and what an organisation writes about its own work
(a proposal, a requirements spec, a technical design, an SOP, a test plan, a roadmap, an insurance
claim, a clinical record).

`categorize` answers it with a cue table first and a model only where the table cannot decide. That
order is the design rather than an optimisation. Typing ten thousand documents through an LLM is
hours of compute to answer a question a regex answers about most of them.

``` python
INVOICE = '''# INVOICE

Invoice No: ACM-2024-0117
Date: 2024-03-01
Payment terms: Net 30
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd

## Line items

| Description | Qty | Unit price | Amount |
|---|---|---|---|
| Widget, steel | 12 | $8.50 | $102.00 |
| Gasket, nitrile | 40 | $1.20 | $48.00 |

Subtotal: $150.00
VAT (20%): $30.00
Total due: $180.00
'''

v.add(INVOICE, 'Acme invoice ACM-2024-0117', source='/inbox/acme-0117.md')
r = v.categorize('/inbox/acme-0117.md', llm='never')
r.doctype, r.score, r.decisive, r.by
```

    ('invoice', 1.0, True, 'cues (ner+regex)')

`score` and `decisive` are the seam. A clear winner needs no model; a two-way tie is exactly the
case worth spending one on, which is what `llm='auto'`, the default, decides for itself. It will use
a model it can find and never start a download to get one. `by` records which leg answered, so a
vault’s types can be audited and re-run selectively.

``` python
v.categorize_all(llm='never')   # everything not typed yet; a failure is recorded, not raised
v.doctypes()                    # the shape of the corpus
```

    {'code': 11, 'documentation': 4, 'invoice': 2, 'paper': 1, 'other': 1}

Notebooks read as code, the README as documentation, the invoice as an invoice, and no model was
loaded to say so. `force=False` makes that cheap to re-run after an ingest: only what arrived since
is looked at. `ner` reads the entity labels off one document, through the same extractor the graph
runs on.

``` python
v.ner('/inbox/acme-0117.md').ents.map(lambda e: (e.label, e.text))[:6]
```

    [('ORG', 'Contoso GmbH'), ('ORG', 'Acme Supplies Ltd'), ('KEYPHRASE', 'Date'), ('KEYPHRASE', 'Payment terms'), ('KEYPHRASE', 'Net'), ('KEYPHRASE', 'Contoso GmbH')]

## Fields, not prose

`extract` reads a whole document and returns a dict. The shapes for the common paperwork are ready,
and a shape you have not declared is one string at the moment you ask.

``` python
from dataclasses import fields
from vishalakshi.extract import SCHEMAS, as_schema

list(SCHEMAS)
```

    ['invoice',
     'purchase_order',
     'quote',
     'receipt',
     'catalogue',
     'contract',
     'resume',
     'paper',
     'meeting_notes',
     'other']

``` python
[f.name for f in fields(as_schema('invoice'))]
```

    ['number',
     'date',
     'due_date',
     'vendor',
     'vendor_tax_id',
     'bill_to',
     'ship_to',
     'currency',
     'subtotal',
     'tax',
     'total',
     'payment_terms',
     'items']

``` python
[f.name for f in fields(as_schema('vendor:str, total:float, due_date:str'))]
```

    ['vendor', 'total', 'due_date']

With no `schema`, the document is categorised first and the shape follows from what it turned out
to be, which is what makes this useful pointed at a folder of mixed paperwork. rishi constrains the
model to the schema (a forced tool call on the hosted and LiteRT backends, a grammar on llama.cpp, a
parsed JSON reply on MLX) so what comes back is the fields you asked for rather than prose about
them. `extract_all(doctype='invoice')` does the same across every invoice in the vault and hands
back one row each, which is a dataframe away from being useful.

``` python
e = v.extract('/inbox/acme-0117.md', chat_kw=dict(backend=Backend.GPU()))      # no schema: the doctype picks one
e.schema, e.fields['total'], e.fields['items']
```

    ('Invoice',
     180.0,
     [{'description': 'Widget, steel',
       'qty': 12.0,
       'unit_price': 8.5,
       'amount': 102.0},
      {'description': 'Gasket, nitrile',
       'qty': 40.0,
       'unit_price': 1.2,
       'amount': 48.0}])

Same call, one argument different, when a bigger model is worth it. The nested `items` list is where a 2B model on CPU struggles and a hosted one does not.

``` python
e = v.extract(r['doc_id'], schema='invoice', model='gpt-4.1-nano')   # needs OPENAI_API_KEY
{k: e.fields[k] for k in ('vendor', 'total', 'payment_terms')} | {'items': e.fields['items']}
```

    {'vendor': 'Acme Supplies Ltd',
     'total': 180,
     'payment_terms': 'Net 30',
     'items': [{'description': 'Widget, steel',
       'qty': 12,
       'unit_price': 8.5,
       'amount': 102},
      {'description': 'Gasket, nitrile',
       'qty': 40,
       'unit_price': 1.2,
       'amount': 48}]}

## Asking about one document, with the vault as context

`ask` retrieves sections from everywhere. Give it `ref=`, or call `ask_doc`, which is the same call
with the name said out loud, and it starts from documents you have already chosen, reads them, and
adds a few retrieved sections behind them. They are sections `[1..n]`, so the citation contract is
the one `ask` already keeps.

Name more than one when the question is a comparison. “What does `extract.py` do that `core.py` does
not” is unanswerable from `extract.py` alone: retrieval will not reliably put the other file in
front of the model, and a model handed one file will confidently describe that file and guess at the
difference. `doc_chars` is the budget for the named documents together, shared between them, because
a context window is a total. Two files at the default 8000 is 4000 characters each.

That default is measured. Through this path on `gemma-4-E2B-it-litert-lm`, 8000 characters (3533
tokens) answers on the first send and 10000 (4265) overflows and needs the retry. Which means two
source files at the local default are truncated to their imports, and a 2B model will say, correctly,
that it cannot tell them apart from what it was given. That is the system working: raise `doc_chars`
and use a model with the window to match, and the same call answers.

Neither document need be in the vault. A path on disk is read straight off it, which is how a
markdown page or a source file gets asked about before it is ever ingested.

``` python
REFS = ['../vishalakshi/extract.py', '../vishalakshi/core.py']   # files on disk, never ingested
Q = 'what does extract.py do that core.py does not?'

a = v.ask_doc(REFS, Q,model=gemma4_e4b,chat_kw=dict(backend=Backend.GPU())) # the default local model: 4000 chars a file
print(a.answer)
# ...and with room for both files, which is what the comparison actually needs:
# v.ask_doc(REFS, Q, doc_chars=60000, model='gpt-5.6-luna')
```

    `extract.py` defines what a document is, the fields inside it, and an answer over the whole of it [1]. It contains definitions for various document types such as `Invoice`, `Receipt`, `Catalogue`,`,`, `Contract`,`, `

And the same question answered as data instead of prose. `schema=` turns any question into a structured response, built at the moment you ask it.

``` python
v.ask_doc('/inbox/acme-0117.md', 'what is owed, to whom, and by when?', model=gemma4_e4b,chat_kw=dict(backend=Backend.GPU()),
          schema='amount:float, currency:str, owed_to:str, due:str').fields
```

    /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/extract.py:584: UserWarning: ValueError on a constrained call for Answer (model neither called the tool nor returned JSON; reply: 'The total due is **$180.00** [1]. It is owed to **Acme Supplies Ltd** [1], and the payment te) — retrying as a JSON reply.
      warnings.warn(f'{type(e).__name__} on a constrained call for {schema.__name__} '

    {'amount': 180.0,
     'currency': 'USD',
     'owed_to': 'Contoso GmbH',
     'due': '2024-04-01'}

## Code, and the two indexes over one tree

A repo is prose and code, and the two want different indexes. `add_tree` splits it: documents to the
vault, source files to kosha, which builds AST chunks, symbol names and a call graph with PageRank.
`grab` routes a directory here, so the one-call path gets it too.

Once a repo is indexed, `context` stops needing to be asked. It appends code sections to what it
retrieves, decided by looking for `.kosha/code.db` on disk rather than by loading anything, so a
question about late chunking pays nothing for a leg it has no use for.

``` python
m=v.index_code('..')                              # this repo; env=True also indexes installed packages
```

    parse files from ..: 100%|██████████| 9/9 [00:00<00:00, 50.10it/s]

<style>
    progress { appearance: none; border: none; border-radius: 4px; width: 300px;
        height: 20px; vertical-align: middle; background: #e0e0e0; }
&#10;    progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; }
    progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; }
    progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; }
&#10;    progress:not([value]) {
        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); }
&#10;    progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted { background: #F44336; }    
&#10;    table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; }
    table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; }
    table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; }
    table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; }
</style>

<style>
    progress { appearance: none; border: none; border-radius: 4px; width: 300px;
        height: 20px; vertical-align: middle; background: #e0e0e0; }
&#10;    progress::-webkit-progress-bar { background: #e0e0e0; border-radius: 4px; }
    progress::-webkit-progress-value { background: #2196F3; border-radius: 4px; }
    progress::-moz-progress-bar { background: #2196F3; border-radius: 4px; }
&#10;    progress:not([value]) {
        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px); }
&#10;    progress.progress-bar-interrupted::-webkit-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-value { background: #F44336; }
    progress.progress-bar-interrupted::-webkit-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted::-moz-progress-bar { background: #F44336; }
    progress.progress-bar-interrupted { background: #F44336; }    
&#10;    table.fastprogress { border-collapse: collapse; margin: 1em 0; font-size: 0.9em; }
    table.fastprogress th, table.fastprogress td { padding: 8px 12px; border: 1px solid #ddd; text-align: left; }
    table.fastprogress thead tr { background: #f8f9fa; font-weight: bold; }
    table.fastprogress tbody tr:nth-of-type(even) { background: #f8f9fa; }
</style>

``` python
c = v.context('where does the entity graph get rebuilt?', sections=3, related=0, code=3, dir='..')
c.code, [r.breadcrumb for r in c.results if r.node_id is None]
```

    (3,
     ['shelf:papers › Attention Is All You Need',
      'repo › ../vishalakshi/core.py:483',
      'grep › README.md:517'])

Those sections are numbered alongside the prose ones and cite like them, so `ask` needed no changes
at all. A code citation just has no `node_id`, because its handle is a `path:line` on disk rather
than something `read()` can open.

`federate` goes wider: the vault, kosha, and ripgrep on the working tree. Three kinds of evidence
and three kinds of blindness. The vault embeds prose; kosha embeds identifiers with a code-trained
model, deliberately, because code embeds badly under a prose encoder; ripgrep embeds nothing and
sees the file as it is on disk right now, including the file nothing has indexed and the edit made a
minute ago. The legs share no vector space, so `federate` fuses their rankings with RRF and never
their distances. `symbol`, `where_to_add` and `grep` are on the [code page](03_code.ipynb).

``` python
L(v.grep('rrf_all', '..', limit=4)).attrgot('where')   # ripgrep, gitignore-aware
```

    ['README.md:538', 'vishalakshi/code.py:12', 'vishalakshi/code.py:128', 'nbs/index.ipynb:1299']

## Watches: keeping it current

An action is the name of an acquisition method, so anything you can file once you can file on a
schedule. `harvest` is one of them, which is what makes a page that renders from an internal JSON
API worth watching: fossick captures the calls the page makes, the vault picks the one carrying
records, follows its pagination, and files each record as its own retrievable section. See the
[acquire page](01_acquire.ipynb) for `apis`, `harvest` and `add_records`.

``` python
v.watch('https://example.com/changelog', action='url', every='6h')
v.watch('late chunking retrieval', action='web', every='1d', n=5)
v.watch('Re-read the eval numbers', action='remind', every='1w')
L(v.watches()).map(lambda w: (w['action'], w['target'][:34], w['every'], w['params']))
```

    [('url', 'https://example.com/changelog', 21600.0, {}), ('web', 'late chunking retrieval', 86400.0, {'n': 5}), ('remind', 'Re-read the eval numbers', 604800.0, {})]

``` python
v.poll() # run everything due; failures are recorded on the row, never raised
```

    [2026-08-12 11:23:35] INFO: Fetched (404) <GET https://example.com/changelog> (referer: https://www.google.com/)
    [2026-08-12 11:23:52] INFO: Fetched (200) <GET https://medium.com/@visrow/what-is-late-chunking-in-rag-how-can-you-improve-your-rag-with-late-chunking-f981a0cb39bb> (referer: https://www.google.com/)
    [2026-08-12 11:23:52] INFO: Fetched (200) <GET https://arxiv.org/pdf/2409.04701> (referer: https://www.google.com/)
    [2026-08-12 11:23:52] INFO: Fetched (200) <GET https://jina.ai/news/late-chunking-in-long-context-embedding-models/> (referer: https://www.google.com/)
    [2026-08-12 11:23:53] INFO: Fetched (200) <GET https://weaviate.io/blog/late-chunking> (referer: https://www.google.com/)
    [2026-08-12 11:23:53] INFO: Fetched (200) <GET https://medium.com/kx-systems/late-chunking-vs-contextual-retrieval-the-math-behind-rags-context-problem-d5a26b9bbd38> (referer: https://www.google.com/)

    {'checked': 3,
     'ran': 3,
     'results': [{'watch_id': '7bba44189449', 'action': 'url', 'target': 'https://example.com/changelog', 'status': 'skipped', 'took': 0.15, 'result': {'url': 'https://example.com/changelog', 'skipped': 'could not read the page (status 404)', 'status': 404}}, {'watch_id': '182c1aff94a3', 'action': 'web', 'target': 'late chunking retrieval', 'status': 'ok', 'took': 18.17, 'result': {'query': 'late chunking retrieval', 'n_found': 5, 'added': [{'doc_id': '5cbfbfdf58728489', 'title': 'Late Chunking: Balancing Precision and Cost in Long Context Retrieval | Weaviate', 'kind': 'web', 'nodes': 13, 'chunks': 39, 'url': 'https://weaviate.io/blog/late-chunking'}, {'doc_id': 'c48bc8d5f932aa2c', 'title': 'arXiv:2409.04701v3 [cs.CL] 7 Jul 2025 LATE CHUNKING: CONTEXTUAL CHUNK EMBED-', 'kind': 'web', 'nodes': 2, 'chunks': 280, 'url': 'https://arxiv.org/pdf/2409.04701'}, {'doc_id': '184c1b13d8f41da3', 'title': 'Late Chunking in Long-Context Embedding Models', 'kind': 'web', 'nodes': 2, 'chunks': 29, 'url': 'https://jina.ai/news/late-chunking-in-long-context-embedding-models/'}, {'doc_id': '094cc044d2ec521b', 'title': 'What is Late Chunking in RAG? How can you improve your RAG with Late Chunking! | by Vishal Mysore | Medium', 'kind': 'web', 'nodes': 5, 'chunks': 10, 'url': 'https://medium.com/@visrow/what-is-late-chunking-in-rag-how-can-you-improve-your-rag-with-late-chunking-f981a0cb39bb'}, {'doc_id': '16f8501b1a83daed', 'title': 'Late Chunking vs Contextual Retrieval: The Math Behind RAG’s Context Problem | by Michael Ryaboy | KX Systems | Medium', 'kind': 'web', 'nodes': 14, 'chunks': 79, 'url': 'https://medium.com/kx-systems/late-chunking-vs-contextual-retrieval-the-math-behind-rags-context-problem-d5a26b9bbd38'}], 'dropped': []}}, {'watch_id': '25f655475f54', 'action': 'remind', 'target': 'Re-read the eval numbers', 'status': 'ok', 'took': 0.0, 'result': {'doc_id': '0bd0b6ea7fd66c83', 'title': 'Re-read the eval numbers', 'kind': 'note', 'nodes': 2, 'chunks': 1}}],
     'next_due': 1786519416.002694}

`poll()` is the tick. Call it from cron, a scheduler, or a frontend button. `remind` writes a note
with no network involved, and one dead URL never stops the loop, because a failure is recorded on the
row rather than raised.

## The rest

Each of these has its own page, and none of them is a decision you have to make to get started.

| page | what is on it |
|----|----|
| [core](00_core.ipynb) | [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault) itself: shelves, `drop_shelf`, `context`, the entity graph, `document` |
| [acquire](01_acquire.ipynb) | `grab`, `url`, `web`, `crawl`, `arxiv`, `pdf`, `youtube`, `github`, `apis`, `harvest`, watches |
| [ask](02_ask.ipynb) | `ask`, `ask_doc`, citations, and the model plumbing |
| [code](03_code.ipynb) | kosha, `symbol`, `where_to_add`, `grep`, `federate` |
| [cli](04_cli.ipynb) | every [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault) method as a command; `--help` generated from the signature |
| [mcp](05_mcp.ipynb) | `vishalakshi-mcp`, and the same methods as tools |
| [extract](06_extract.ipynb) | `categorize`, `extract`, `extract_all`, the schemas |
| [concepts](07_concepts.ipynb) | doctypes, `reshelf`, and what the vault decides about a document |
| [pii](09_pii.ipynb) | the patterns, their checksums, [`redact`](https://vedicreader.github.io/vishalakshi/pii.html#redact), and what gates an answer |

``` sh
vishalakshi grab https://example.com/post     # or a file, a directory, an arXiv id, a YouTube URL
vishalakshi ask "why does late chunking help"
```

`$VISHALAKSHI_VAULT` picks the vault file, `$VISHALAKSHI_MODEL` the model `ask` uses,
`$VISHALAKSHI_PII_MODEL` the local one it falls back to when the sections are private,
`$VISHALAKSHI_GPU=0` puts LiteRT on the CPU, and `$VISHALAKSHI_OFFLINE` forces the hashing encoder. `vishalakshi-mcp` exposes the same methods to any
MCP client:

``` json
{"mcpServers": {"vishalakshi": {"command": "vishalakshi-mcp",
                                "env": {"VISHALAKSHI_VAULT": "~/.vishalakshi/vault.db"}}}}
```

## Development

The notebooks in `nbs/` are the source; the modules are generated.

``` sh
pip install -e '.[all]'
nbdev-prepare
```
