Metadata-Version: 2.4
Name: vishalakshi
Version: 0.1.2
Summary: one vault for everything you read: web, papers, video, files and code, searchable together and answerable by a local or hosted model
Project-URL: Repository, https://github.com/vedicreader/vishalakshi
Project-URL: Documentation, https://vedicreader.github.io/vishalakshi/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document classification,information extraction,knowledge base,memory,nbdev,ner,rag,research,semantic search,structured output
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.12
Requires-Dist: fastcore>=1.12.31
Requires-Dist: fossick>=0.1.7
Requires-Dist: koshas>=0.0.36
Requires-Dist: litesearch[sanskrit]>=0.1.25
Requires-Dist: mcp
Requires-Dist: numpy>=1.26
Requires-Dist: pyskills>=0.0.27
Requires-Dist: rgapi
Requires-Dist: rishi[litert,remote]>=0.1.2
Provides-Extra: test
Requires-Dist: diskcache; extra == 'test'
Description-Content-Type: text/markdown

# vishalakshi


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

## Install

``` sh
pip install 'vishalakshi[all]'   # + rishi to answer, + kosha & rgapi for code, + mcp for the server
pip install vishalakshi          # vault + acquisition only; no LLM, no MCP
```

## The loop

Everything in this section runs. The corpus is this repository’s own documentation and source, so
the page is reproducible from a clone, and the vault is a throwaway file rather than your real one.
[`Vault()`](https://vedicreader.github.io/vishalakshi/core.html#vault) with no argument uses `~/.vishalakshi/vault.db`.

``` python
from tempfile import mkdtemp
from vishalakshi import Vault

v = Vault(Path(mkdtemp())/'vault.db')
v.enc.note
```

    'minishlab/potion-multilingual-128M (256d, float16, model2vec)'

Ingestion is one call. `add` takes a directory, a file, or text, and routes on which it got.

``` python
v.add('..')                       # README.md and every notebook under nbs/
v.note('federate fuses the legs by rank because they share no vector space: the vault embeds '
       'prose, kosha embeds identifiers, ripgrep embeds nothing.', tags=['retrieval', 'design'])
```

    {'doc_id': 'ed3044d347f8f257',
     'title': 'federate fuses the legs by rank because they share no vector space: the vault em',
     'kind': 'note',
     'nodes': 2,
     'chunks': 1}

Then reach outward. `grab` routes on what the target is: an arXiv id, a YouTube link, a GitHub
repo, a PDF, a local file, a directory. It is the one call a CLI or an agent needs.

``` python
v.grab('1706.03762')                                       # an arXiv id: metadata + abstract
v.pdf('https://arxiv.org/pdf/1706.03762',                  # and the full paper, page by page
      title='Attention Is All You Need')
```

    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream
    Dictionary used where Stream expected, treating as empty stream

    {'doc_id': '47e4bf9c7b60a727',
     'title': 'Attention Is All You Need',
     'kind': 'pdf',
     'nodes': 5,
     'chunks': 102}

Now ask it something. The model is rishi’s own default, `gemma4_e2b`: LiteRT, CPU, no API key.
Naming a model is naming it to rishi, so an id, a path or a `runtime/...` prefix all work, and
`chat_kw=` reaches the rest of rishi’s constructor. There is no model registry here to fall out of
date.

The defaults hand the model four sections at 1500 characters each. That is the point of retrieving
*sections* rather than documents: the answer needs the paragraphs that bear on the question, not the
corpus. A pointed context is why a 2B model on a laptop can answer at all.

``` python
from rishi.litert import gemma4_e2b

r = v.ask('why are rankings fused instead of distances?', model=gemma4_e2b)
print(r.model, '·', r.runtime)
print(r.answer)
```

    litert-community/gemma-4-E2B-it-litert-lm · litert
    The vault embeds prose, kosha embeds identifiers, and ripgrep embeds nothing, which means legs cannot be merged by distance [2]. Reciprocal Rank Fusion (RRF) only needs each leg's ordering, which is what survives a change in encoder, and it is the same mechanism LiteSearch already uses to combine FTS with vectors [2]. Each leg is tried independently to report what it contributed or why it did not [2].

Every `[n]` in the answer resolves to a `node_id` you can read. That round trip is the point: a claim you can check against the text it came from.

``` python
for c in r.cited: print(c['n'], c['breadcrumb'])
print()
print(v.read(r.cited[0]['node_id'])['text'][:400])   # the exact text behind the claim
```

    2 03 code › Fusing legs that share no vector space

    The vault embeds prose, kosha embeds identifiers, ripgrep embeds nothing, so the legs cannot be
    merged by distance. Reciprocal Rank Fusion needs only each leg's *ordering* — which is exactly
    what survives a change of encoder — and it is the same mechanism litesearch already uses to
    combine FTS with vectors. Each leg is tried independently: `legs` reports what each contributed,
    or why it did not.

## What is decided, and what is yours

The retrieval defaults are litesearch’s, and they are measured rather than chosen: `evals/` runs 120
known-item queries per genre over three corpora and scores section-level MRR. You inherit the whole
ladder by opening a vault.

| decided for you | why |
|----|----|
| 512-character chunks | +0.06 to +0.12 weighted MRR over page-sized |
| `pre()` on the keyword leg | +0.016 to +0.093 |
| HNSW vector index | -0.005 quality for a large speedup |
| a document tree, always | ranking is a wash, and `toc`/`read`/`sections`/`context` come free |
| one static encoder, 256d, float16 | the spread across four encoders is 0.018 to 0.046, and the static one wins a genre at ~1,700x cheaper indexing |
| Devanagari and IAST fold to the same token | 1.000 verse recall for every encoder tested, from the tokenizer rather than the embedding |

Four things are yours, and none of them is required to get an answer.

| yours | what it costs |
|----|----|
| `rerank=True` on `search`, `sections`, `context` | +0.026 to +0.077 weighted MRR, positive in all twelve paired cells, at roughly 10x the query latency |
| `shelf(name)` | a partition, so two corpora stop diluting each other’s ranking |
| `llm=` on `categorize` and `extract` | the cue table answers most documents for free; a model is for the ties |
| `db.graph_search` | the graph leg by name, for bridge queries |

The graph leg is off for ranking and stays off. On ordinary known-item questions it costs 0.070 to
0.160 weighted MRR, negative in every genre and flavour measured. It wins only when the answer
shares no word with the question. `connect()` is still worth running, for the reason below.

Sanskrit is the one place something switches itself on. Ingest a Sanskrit file and the vault
re-registers litesearch’s readers with vidyut lemmas and Monier-Williams glosses, because putting
the English behind the Sanskrit into the index beat changing the encoder when it was measured on
this vault’s own shelf: a static encoder with glosses beat a 300M ONNX transformer without them. It
is an ~83 MB download, paid once, only by someone actually reading Sanskrit.

## Everything in one corpus

Every document carries a `kind`: `web`, `pdf`, `arxiv`, `youtube`, `file`, `code`, `data`, `note`,
or whatever litesearch’s parser called the file. Filter when you want to, don’t when you don’t.

``` python
v.stats()
```

    {'docs': 16,
     'nodes': 134,
     'chunks': 691,
     'encoder': 'model2vec',
     'entities': 0,
     'path': '/var/folders/kg/9vdw4mdd1fs58svgh4k1qhr09x7dqh/T/tmpxjla5hox/vault.db',
     'by_kind': {'notebook': 10, 'md': 3, 'txt': 1, 'pdf': 1, 'note': 1}}

``` python
len(v.search('rank fusion')), len(v.search('rank fusion', kind='note')), len(v.search('rank fusion', kind='md,notebook'))
```

    (10, 1, 10)

The filter is a SQL `WHERE` pushed into the search, not a pass over the results afterwards, so a
narrow filter over a large vault still returns a full page of hits.

Notes are ordinary documents, deliberately. The graph, the clusters and `context()` all see them for
free, so what you concluded about a corpus comes back next to the evidence you concluded it from.

``` python
L(v.sources()).map(lambda d: (d['kind'], d['title'], d['source']))[:6]
```

    [('pdf', 'Attention Is All You Need', 'https://arxiv.org/pdf/1706.03762'), ('md', 'CHANGELOG', '../CHANGELOG.md'), ('md', 'README', '../README.md'), ('notebook', '00 core', '../nbs/00_core.ipynb'), ('notebook', '01 acquire', '../nbs/01_acquire.ipynb'), ('notebook', '02 ask', '../nbs/02_ask.ipynb')]

## When the graph and the clusters pay off

`connect()` builds an entity graph over everything in the vault, and `map()` reads the topics it
persisted. Neither is retrieval. The graph leg loses on ordinary queries, which is why it is not
wired into `search`.

What they answer is the question you ask *before* you know what to search for. `map()` is the shape
of what you have collected, which is how you notice that four months of reading turned into three
subjects rather than the one you thought. The graph is for reaching across shelves and across
documents that never share a word: a bridge query where the paper and the invoice both mention a
company neither of them names in the same way. Run `connect()` after a batch of ingests rather than
inside each one, since it rebuilds over the whole vault.
`topic_tree` puts the documents under each label, with the number of chunks each contributes.

That second number is the one worth reading. A topic carried by a single document is usually that
document talking to itself, and a label to distrust. A topic spread across five is a thread actually
running through the corpus, and the place to start reading. It is also how you notice that two
labels you thought were separate subjects are the same source seen twice.

Three queries and no clustering: `connect()` already persisted the topic nodes, so this only pivots
the mentions back onto documents.

``` python
v.connect()
```

    {'entities': 3860,
     'mentions': 7314,
     'edges': 1987,
     'windows': 2305,
     'resolved': {'merged': 1058,
      'by_ann': 593,
      'by_lexical': 465,
      'edges': 1695,
      'entities': 3860,
      'resolvable': 3860,
      'canonical': 2802},
     'topics': 136,
     'method': 'knn'}

``` python
v.show_topics(limit=5, docs=3)      # v.topic_tree() returns the same thing as data
```

    ents, cues, needs, money                     (8 chunks, 1 docs)
      `- 06 extract                                  8
    strip, meta, fetched_at, parts               (8 chunks, 4 docs)
      |- 01 acquire                                  3
      |- 00 core                                     2
      `- 02 ask                                      2
    doctype, save, schema, chat_kw               (8 chunks, 1 docs)
      `- 06 extract                                  8
    docs, doc_id, doctype, self                  (8 chunks, 4 docs)
      |- 06 extract                                  4
      |- 02 ask                                      2
      `- 00 core                                     1
    dims, hash, mk_encoder, test_eq              (8 chunks, 1 docs)
      `- 00 core                                     8

## What each document is, and what is inside it

`kind` says how a document arrived. What it *is* (an invoice, a price list, a contract, a paper, a
transcript, a source file) is a different question, and one worth answering: a vault that knows
which of its documents are invoices can hand you their totals as a table.

`categorize` answers it with a cue table first and a model only where the table cannot decide. That
order is the design rather than an optimisation. Typing ten thousand documents through an LLM is
hours of compute to answer a question a regex answers about most of them.

``` python
INVOICE = '''# INVOICE

Invoice No: ACM-2024-0117
Date: 2024-03-01
Payment terms: Net 30
Bill to: Contoso GmbH, Berlin
From: Acme Supplies Ltd

## Line items

| Description | Qty | Unit price | Amount |
|---|---|---|---|
| Widget, steel | 12 | $8.50 | $102.00 |
| Gasket, nitrile | 40 | $1.20 | $48.00 |

Subtotal: $150.00
VAT (20%): $30.00
Total due: $180.00
'''

v.add(INVOICE, 'Acme invoice ACM-2024-0117', source='/inbox/acme-0117.md')
r = v.categorize('/inbox/acme-0117.md', llm='never')
r.doctype, r.score, r.decisive, r.by
```

    ('invoice', 1.0, True, 'cues (ner+regex)')

`score` and `decisive` are the seam. A clear winner needs no model; a two-way tie is exactly the
case worth spending one on, which is what `llm='auto'`, the default, decides for itself. It will use
a model it can find and never start a download to get one. `by` records which leg answered, so a
vault’s types can be audited and re-run selectively.

``` python
v.categorize_all(llm='never')   # everything not typed yet; a failure is recorded, not raised
v.doctypes()                    # the shape of the corpus
```

    {'code': 10, 'documentation': 4, 'invoice': 1, 'paper': 1, 'other': 1}

Notebooks read as code, the README as documentation, the invoice as an invoice, and no model was
loaded to say so. `force=False` makes that cheap to re-run after an ingest: only what arrived since
is looked at. `ner` reads the entity labels off one document, through the same extractor the graph
runs on.

``` python
v.ner('/inbox/acme-0117.md').ents.map(lambda e: (e.label, e.text))[:6]
```

    [('ORG', 'Contoso GmbH'), ('ORG', 'Acme Supplies Ltd'), ('KEYPHRASE', 'Date'), ('KEYPHRASE', 'Payment terms'), ('KEYPHRASE', 'Net'), ('KEYPHRASE', 'Contoso GmbH')]

## Fields, not prose

`extract` reads a whole document and returns a dict. The shapes for the common paperwork are ready,
and a shape you have not declared is one string at the moment you ask.

``` python
from dataclasses import fields
from vishalakshi.extract import SCHEMAS, as_schema

list(SCHEMAS)
```

    ['invoice',
     'purchase_order',
     'quote',
     'receipt',
     'catalogue',
     'contract',
     'resume',
     'paper',
     'meeting_notes',
     'other']

``` python
[f.name for f in fields(as_schema('invoice'))]
```

    ['number',
     'date',
     'due_date',
     'vendor',
     'vendor_tax_id',
     'bill_to',
     'ship_to',
     'currency',
     'subtotal',
     'tax',
     'total',
     'payment_terms',
     'items']

``` python
[f.name for f in fields(as_schema('vendor:str, total:float, due_date:str'))]
```

    ['vendor', 'total', 'due_date']

With no `schema`, the document is categorised first and the shape follows from what it turned out
to be, which is what makes this useful pointed at a folder of mixed paperwork. rishi constrains the
model to the schema (a forced tool call on the hosted and LiteRT backends, a grammar on llama.cpp, a
parsed JSON reply on MLX) so what comes back is the fields you asked for rather than prose about
them. `extract_all(doctype='invoice')` does the same across every invoice in the vault and hands
back one row each, which is a dataframe away from being useful.

``` python
e = v.extract('/inbox/acme-0117.md', chat_kw=dict(backend=Backend.GPU()))      # no schema: the doctype picks one
e.schema, e.fields['total'], e.fields['items']
```

    ('Invoice',
     180.0,
     [{'description': 'Widget, steel',
       'qty': 12.0,
       'unit_price': 8.5,
       'amount': 102.0},
      {'description': 'Gasket, nitrile',
       'qty': 40.0,
       'unit_price': 1.2,
       'amount': 48.0}])

Same call, one argument different, when a bigger model is worth it. The nested `items` list is where a 2B model on CPU struggles and a hosted one does not.

``` python
e = v.extract(r['doc_id'], schema='invoice', model='gpt-4.1-nano')   # needs OPENAI_API_KEY
{k: e.fields[k] for k in ('vendor', 'total', 'payment_terms')} | {'items': e.fields['items']}
```

    {'vendor': 'Acme Supplies Ltd',
     'total': 180,
     'payment_terms': 'Net 30',
     'items': [{'description': 'Widget, steel',
       'qty': 12,
       'unit_price': 8.5,
       'amount': 102},
      {'description': 'Gasket, nitrile',
       'qty': 40,
       'unit_price': 1.2,
       'amount': 48}]}

## Asking about one document, with the vault as context

`ask` retrieves sections from everywhere. Give it `ref=`, or call `ask_doc`, which is the same call
with the name said out loud, and it starts from documents you have already chosen, reads them, and
adds a few retrieved sections behind them. They are sections `[1..n]`, so the citation contract is
the one `ask` already keeps.

Name more than one when the question is a comparison. “What does `extract.py` do that `core.py` does
not” is unanswerable from `extract.py` alone: retrieval will not reliably put the other file in
front of the model, and a model handed one file will confidently describe that file and guess at the
difference. `doc_chars` is the budget for the named documents together, shared between them, because
a context window is a total. Two files at the default 8000 is 4000 characters each.

That default is measured. Through this path on `gemma-4-E2B-it-litert-lm`, 8000 characters (3533
tokens) answers on the first send and 10000 (4265) overflows and needs the retry. Which means two
source files at the local default are truncated to their imports, and a 2B model will say, correctly,
that it cannot tell them apart from what it was given. That is the system working: raise `doc_chars`
and use a model with the window to match, and the same call answers.

Neither document need be in the vault. A path on disk is read straight off it, which is how a
markdown page or a source file gets asked about before it is ever ingested.

``` python
REFS = ['../vishalakshi/extract.py', '../vishalakshi/core.py']   # files on disk, never ingested
Q = 'what does extract.py do that core.py does not?'

a = v.ask_doc(REFS, Q,model=gemma4_e4b,chat_kw=dict(backend=Backend.GPU()))                                   # the default local model: 4000 chars a file
print(a.answer)
# ...and with room for both files, which is what the comparison actually needs:
# v.ask_doc(REFS, Q, doc_chars=60000, model='gpt-5.6-luna')
```

    `extract.py` defines what a document is, the fields inside it, and an answer over the whole of it [1]. It contains definitions for various document types such as `Invoice`, `Receipt`, `Catalogue`, ` `Contract`, ` ` `Resume`, ` `Paper`, ` `MeetingNotes`, and `Summary` [1].

    The `core.py

And the same question answered as data instead of prose. `schema=` turns any question into a structured response, built at the moment you ask it.

``` python
v.ask_doc('/inbox/acme-0117.md', 'what is owed, to whom, and by when?', model=gemma4_e4b,chat_kw=dict(backend=Backend.GPU()),
          schema='amount:float, currency:str, owed_to:str, due:str').fields
```

    /Users/71293/code/personal/orgs/vishalakshi/vishalakshi/extract.py:527: UserWarning: ValueError on a constrained call for Answer (model neither called the tool nor returned JSON; reply: 'The total amount owed is **$180.00** [1]. This is owed to **Acme Supplies Ltd** [1, 4]. The p) — retrying as a JSON reply.
      warnings.warn(f'{type(e).__name__} on a constrained call for {schema.__name__} '

    {'amount': 180.0,
     'currency': 'USD',
     'owed_to': 'Acme Supplies Ltd',
     'due': 'Net 30'}

## Code, and the two indexes over one tree

A repo is prose and code, and the two want different indexes. `add_tree` splits it: documents to the
vault, source files to kosha, which builds AST chunks, symbol names and a call graph with PageRank.
`grab` routes a directory here, so the one-call path gets it too.

Once a repo is indexed, `context` stops needing to be asked. It appends code sections to what it
retrieves, decided by looking for `.kosha/code.db` on disk rather than by loading anything, so a
question about late chunking pays nothing for a leg it has no use for.

``` python
m=v.index_code('..')                              # this repo; env=True also indexes installed packages
```

    █ |----------------------------------------| 0.00% [0/3 00:00<?]█ |----------------------------------------| 0.00% [0/1 00:00<?] |████████████████████████████████████████| 100.00% [1/1 00:00<00:00]                                                                      |█████████████---------------------------| 33.33% [1/3 00:00<00:00] |██████████████████████████--------------| 66.67% [2/3 00:00<00:00] |████████████████████████████████████████| 100.00% [3/3 00:00<00:00]                                                                     

    parse files from ..:   0%|          | 0/8 [00:00<?, ?it/s]parse files from ..:  25%|##5       | 2/8 [00:00<00:00, 10.76it/s]parse files from ..: 100%|##########| 8/8 [00:00<00:00, 34.50it/s]

``` python
c = v.context('where does the entity graph get rebuilt?', sections=3, related=0, code=3, dir='..')
c.code, [r.breadcrumb for r in c.results if r.node_id is None]
```

    (3,
     ['shelf:papers › Attention Is All You Need',
      'repo › ../vishalakshi/core.py:483',
      'grep › README.md:457'])

Those sections are numbered alongside the prose ones and cite like them, so `ask` needed no changes
at all. A code citation just has no `node_id`, because its handle is a `path:line` on disk rather
than something `read()` can open.

`federate` goes wider: the vault, kosha, and ripgrep on the working tree. Three kinds of evidence
and three kinds of blindness. The vault embeds prose; kosha embeds identifiers with a code-trained
model, deliberately, because code embeds badly under a prose encoder; ripgrep embeds nothing and
sees the file as it is on disk right now, including the file nothing has indexed and the edit made a
minute ago. The legs share no vector space, so `federate` fuses their rankings with RRF and never
their distances. `symbol`, `where_to_add` and `grep` are on the [code page](03_code.ipynb).

``` python
L(v.grep('rrf_all', '..', limit=4)).attrgot('where')   # ripgrep, gitignore-aware
```

    ['README.md:204', 'README.md:478', 'nbs/index.ipynb:443', 'nbs/index.ipynb:1026']

## Watches: keeping it current

An action is the name of an acquisition method, so anything you can file once you can file on a
schedule. `harvest` is one of them, which is what makes a page that renders from an internal JSON
API worth watching: fossick captures the calls the page makes, the vault picks the one carrying
records, follows its pagination, and files each record as its own retrievable section. See the
[acquire page](01_acquire.ipynb) for `apis`, `harvest` and `add_records`.

``` python
v.watch('https://example.com/changelog', action='url',     every='6h')
v.watch('late chunking retrieval',       action='web',     every='1d', n=5)
v.watch('Re-read the eval numbers',      action='remind',  every='1w')

L(v.watches()).map(lambda w: (w['action'], w['target'][:34], w['every'], w['params']))
```

    [('url', 'https://example.com/changelog', 21600.0, {}), ('web', 'late chunking retrieval', 86400.0, {'n': 5}), ('remind', 'Re-read the eval numbers', 604800.0, {})]

``` python
v.poll()      # run everything due; failures are recorded on the row, never raised
```

    {'checked': 3,
     'ran': 3,
     'results': [{'watch_id': 'cf1fe0a11b21', 'action': 'url', 'target': 'https://example.com/changelog', 'status': 'skipped', 'took': 0.12, 'result': {'url': 'https://example.com/changelog', 'skipped': 'could not read the page (status 404)', 'status': 404}}, {'watch_id': 'ed8c2c2c9c7a', 'action': 'web', 'target': 'late chunking retrieval', 'status': 'ok', 'took': 3.57, 'result': {'query': 'late chunking retrieval', 'n_found': 5, 'added': [{'doc_id': '5ff9f204a961b1d1', 'title': 'arxiv.org › abs › 2409[2409.04701] Late Chunking: Contextual Chunk Embeddings Using...arxiv.org › html › 2409Late Chunki', 'kind': 'web', 'nodes': 3, 'chunks': 6, 'url': 'https://arxiv.org/abs/2409.04701'}, {'doc_id': '504a7c1f1326cc02', 'title': 'weaviate.io › blog › late-chunkingLate Chunking: Balancing Precision and Cost in Long Context...medium.com › @visrow › c', 'kind': 'web', 'nodes': 13, 'chunks': 39, 'url': 'https://weaviate.io/blog/late-chunking'}, {'doc_id': '094cc044d2ec521b', 'title': 'What is Late Chunking in RAG? How can you improve your RAG with Late Chunking! | by Vishal Mysore | Medium', 'kind': 'web', 'nodes': 5, 'chunks': 10, 'url': 'https://medium.com/@visrow/what-is-late-chunking-in-rag-how-can-you-improve-your-rag-with-late-chunking-f981a0cb39bb'}, {'doc_id': '184c1b13d8f41da3', 'title': 'Late Chunking in Long-Context Embedding Models', 'kind': 'web', 'nodes': 2, 'chunks': 29, 'url': 'https://jina.ai/news/late-chunking-in-long-context-embedding-models/'}, {'doc_id': '14bf717429612609', 'title': 'GitHub - jina-ai/late-chunking: Code for explaining and evaluating late chunking (chunked pooling) · GitHub', 'kind': 'web', 'nodes': 6, 'chunks': 19, 'url': 'https://github.com/jina-ai/late-chunking'}], 'dropped': []}}, {'watch_id': '26cad856aaa8', 'action': 'remind', 'target': 'Re-read the eval numbers', 'status': 'ok', 'took': 0.0, 'result': {'doc_id': '71b411b34d0b6973', 'title': 'Re-read the eval numbers', 'kind': 'note', 'nodes': 2, 'chunks': 1}}],
     'next_due': 1786427811.255703}

`poll()` is the tick. Call it from cron, a scheduler, or a frontend button. `remind` writes a note
with no network involved, and one dead URL never stops the loop, because a failure is recorded on the
row rather than raised.

## The rest

Each of these has its own page, and none of them is a decision you have to make to get started.

| page | what is on it |
|----|----|
| [core](00_core.ipynb) | [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault) itself: shelves, `drop_shelf`, `context`, the entity graph, `document` |
| [acquire](01_acquire.ipynb) | `grab`, `url`, `web`, `crawl`, `arxiv`, `pdf`, `youtube`, `github`, `apis`, `harvest`, watches |
| [ask](02_ask.ipynb) | `ask`, `ask_doc`, citations, and the model plumbing |
| [code](03_code.ipynb) | kosha, `symbol`, `where_to_add`, `grep`, `federate` |
| [cli](04_cli.ipynb) | every [`Vault`](https://vedicreader.github.io/vishalakshi/core.html#vault) method as a command; `--help` generated from the signature |
| [mcp](05_mcp.ipynb) | `vishalakshi-mcp`, and the same methods as tools |
| [extract](06_extract.ipynb) | `categorize`, `extract`, `extract_all`, the schemas |
| [concepts](07_concepts.ipynb) | doctypes, `reshelf`, and what the vault decides about a document |

``` sh
vishalakshi grab https://example.com/post     # or a file, a directory, an arXiv id, a YouTube URL
vishalakshi ask "why does late chunking help"
```

`$VISHALAKSHI_VAULT` picks the vault file, `$VISHALAKSHI_MODEL` the model `ask` uses, and
`$VISHALAKSHI_OFFLINE` forces the hashing encoder. `vishalakshi-mcp` exposes the same methods to any
MCP client:

``` json
{"mcpServers": {"vishalakshi": {"command": "vishalakshi-mcp",
                                "env": {"VISHALAKSHI_VAULT": "~/.vishalakshi/vault.db"}}}}
```

## Development

The notebooks in `nbs/` are the source; the modules are generated.

``` sh
pip install -e '.[all]'
nbdev-prepare
```
