Metadata-Version: 2.4
Name: fihrist
Version: 0.1.1
Summary: Use the World Lattice to classify books by what they are about, with prompt building, hash verification, and model-output validation.
Project-URL: Homepage, https://github.com/bk2022-ui/lattice
Project-URL: Source, https://github.com/bk2022-ui/lattice
Project-URL: Issues, https://github.com/bk2022-ui/lattice/issues
Project-URL: World Lattice, https://huggingface.co/datasets/bk2022/world-lattice
Project-URL: Corpus-425, https://huggingface.co/datasets/bk2022/corpus-425
Author: Bharat Khandelwal
License: Apache-2.0
License-File: LICENSE
License-File: LICENSE-DATA
Keywords: bias,books,classification,faceted-classification,libraries,llm,taxonomy
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Education
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Sociology :: History
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# Fihrist

**Use the World Lattice to classify books by what they are about, not where their authors are from.**

```bash
pip install fihrist
```

Fihrist is the Python package for the
[World Lattice](https://huggingface.co/datasets/bk2022/world-lattice): a 3,660-leaf,
ten-axis faceted classification system for books.

It ships the Lattice artifact, verifies its hash, builds the prompts for enrichment
and slotting, and validates model replies. It makes no network calls and never sees an
API key. You choose the model; Fihrist gives you the frame and checks the answer.

## The problem this exists to solve

Ask a language model to sort a library and it may file an Indian doctor's **genetics**
books under "India." It may file a Japanese **lean-manufacturing** book under "Japan"
and lose Operations entirely. Both of those are real, from the project this package
came out of, and both were caught by a human rather than the machine.

That is not a quirk of one model. It is a century-old bias the models learned from us. Open
Dewey Decimal at religion:

| | Dewey range |
|---|---|
| Christianity | **200–289** |
| *All* Indic religions | **294** — Hinduism is *294.5* |

Ninety classes against one decimal place. Library of Congress does the same thing
structurally: the Americas get two top-level classes; Asia and Africa share one. Train on a
century of text organised that way and a model inherits the shape.

The bias shows up as two distinct failures, and you will hit both:

| | Failure | What it costs you |
|---|---|---|
| **Identity magnet** | the author's origin substitutes for the book's content | a **wrong** home |
| **Completeness** | one obvious home hides the others | a **missing** home |

The second is the one that quietly ruins a catalogue. A book on Toyota's production system is
about operations *and* Japan. A novel can be literature *and* psychology. Force one label and
you throw away most of what you knew.

## What Fihrist does about it

The World Lattice is built so no tradition sits at the centre as the unmarked default:
Christianity 12 leaves, Islam 12, Hinduism 11, Buddhism 10; Ayurveda 12 and
Traditional Chinese Medicine 10 beside *Western humoral / Hippocratic-Galenic
medicine* 7. The West is named as a tradition rather than assumed as the baseline.
Every leaf carries a **definition**, so a placement can be argued against a written
scope note instead of a label.

Because the axes are independent, a book attaches to as many as its content earns. In practice
most books touch five to eight. Nothing has to lose for something else to win.

Fihrist does the deterministic half: it **builds the prompt that asks a model to place
the book, and checks the answer that comes back.** Models invent plausible leaves.
Fihrist drops anything not in the frame, repairs a leaf filed under the wrong parent,
and tells you when a batch came back silently empty.

Named for the *Kitāb al-Fihrist* (Baghdad, 987 CE), a bookseller's attempt to index every book
then known, across every language he could reach.

---

## What you need before you start

**Fihrist takes an enriched book record, not a title.**

A title and an author are not enough, and that is the whole point rather than a missing feature.
The lattice slots on what a book *contains*. Feed it a bare title and the only thing a model can
classify on is the author's name and origin — which is precisely the bias the artifact exists to
remove.

So the input is a record carrying **idea units**: the handful of things the book is actually
about. Step 2 below shows how to get them.

## What Fihrist does and does not do

Fihrist makes **no network calls** and **never sees an API key**. It builds prompts and checks
answers. The model call is yours, on your account, with your choice of provider.

| Fihrist owns | You own |
|---|---|
| the artifact, verified against its sha256 on load | the API key |
| leaf and parent lookup, the embedded parity audits | the model call |
| building the research prompt | running it |
| building the slot prompt | running it |
| validating the reply — invented leaves are dropped | |

Reads as a limitation for about ten seconds, and as a feature forever: no key, no telemetry, no
surprise bill, and it works on a plane.
---

## 1 — Inspect the artifact

Offline, immediate, free.

```python
import fihrist

lat = fihrist.load()              # hash-verified against the shipped sha256
len(lat)                          # 3660
lat.axes                          # ['Discipline', 'Place', 'Time', 'Form', 'Genre', ...]
len(lat.leaves("Discipline"))     # 1246

lat.find("Meiji")                 # [Leaf('Modern Japan (Meiji–Shōwa)', axis='Discipline', ...)]
lat.parents_of("Modern Japan (Meiji–Shōwa)", "Discipline")
# ['Japanese history']

lat.substantive_audits            # the 6 written parity audits, carried inside the file
lat.audited_axes                  # which axes they cover
```

The artifact carries its own audits. You do not have to take the parity claim on trust — open
`lat.substantive_audits` and read what was checked, including the one that returned **FAIL** on the
Place axis.

Six of the ten axes have a written audit. `Time`, `Thing/Matter` and `Form` carry a stub rather than
a finding and should be treated as unaudited; `Genre` has none. `lat.audited_axes` names the six.

From the shell:

```bash
fihrist verify
fihrist axes
fihrist leaves --axis Discipline --grep literature
```

## 2 — Enrich a book

Fihrist writes the research prompt. You run it against any model that can search the web.

```python
prompt = fihrist.enrich_prompt("I Am a Cat", "Natsume Sōseki")
record = fihrist.parse_enrichment(your_model(prompt))

record["ideaUnits"]
# [{'id': 'the-outsider-narrator-sees-clearest-1',
#   'title': 'The Outsider Narrator Sees Clearest',
#   'tags': ['narration', 'satire', 'detachment']}, ...]
```

The instruction body is frozen and byte-identical for every book — only the assignment block at the
end changes. That constancy is deliberate. A prompt that said more about a Japanese novel than a
French one would manufacture the very difference the artifact exists to measure, then hand it to the
slotter as though the lattice had caused it.

## 3 — Slot it

```python
prompt = fihrist.slot_prompt([record])
results = fihrist.validate(your_model(prompt), [record])

results[0].attached[0]
```

```json
{"axis": "Discipline", "parent": "Japanese history", "leaf": "Modern Japan (Meiji–Shōwa)",
 "confidence": "stretch", "culture": "Japanese", "status": "valid"}
```

Batch as many books into one `slot_prompt` call as your context window allows. The lattice index is
rendered first and identically every time, so a prompt-caching backend caches it across a whole run.

### Validation is not a formality

```python
fihrist.summarise(results)
# {'books': 1, 'attachments': 7, 'rejectedAttachments': 1,
#  'booksWithNoAttachment': 0, 'leavesPerBook': 7.0, 'byAxis': {...}}
```

Three things worth knowing:

- **Invented leaves are dropped.** Models produce plausible leaves in the artifact's own house style.
  An invented one that survives into your results is worse than a missing one, because it looks like
  evidence. Anything not on the lattice goes to `rejected` with a reason, never into `attached`.
- **A wrong parent is repaired, and the repair is recorded** in `parentRepaired`. Polyhierarchy means
  a leaf can hang from several parents; any true one is accepted.
- **Check `booksWithNoAttachment` first.** A batch that comes back structurally valid and
  substantively empty is the quietest way a run fails — the JSON parses, nothing raises, and the
  numbers are simply thin. That counter is where it shows.

## From the shell

```bash
fihrist verify                                    # hash check
fihrist axes                                      # the 10 axes
fihrist leaves --axis Genre --grep thriller
fihrist prompt enrich --title "I Am a Cat" --author "Natsume Sōseki"
fihrist prompt slot books.json > prompt.txt
fihrist validate response.json --books books.json --out slotted.json
```

---

## The claim, and its limits

Seven engines — Claude, GPT, DeepSeek, and local Qwen models from 30B to 235B — slotted the same
1,232 books onto this lattice. **All of them top-ranked India, China and Japan on the Place axis.** A
biased engine buries those. None did. The correction lives in the artifact rather than in a prompt or
a model, so swapping the model does not bring the bias back.

What is **not** established: the 425-book companion corpus should be read as a test
dataset and model-comparison record, not as a final population-level proof. It is
published because the Lattice is meant to be inspected on books beyond the library it
was built beside. See the corpus dataset for the records and known limitations.

Two independent things are worth keeping apart. **Audit coverage:** six axes carry a written parity
audit — Discipline, Place, Agent/Entity, Action/Process, Language, Ethnic/National-group. The other
four do not, and `Place`'s own audit records a verdict of **FAIL** with the remediation that
followed. **Slotting maturity:** three axes — `Agent/Entity`, `Action/Process`, `Thing/Matter` — are
provisional as slotting targets and marked as such in `fihrist.PROVISIONAL_AXES`.

## The guard rules

`fihrist.GUARDS` is the anti-bias correction, carried verbatim from the run that produced the
published results. **Rewording it invalidates every parity claim attached to this artifact.** If you
need different behaviour, write your own prompt rather than editing those lines, and do not report
the result as a Fihrist placement.

## Cost

Fihrist itself is free and offline. Steps 2 and 3 are your own model calls — roughly one
search-enabled call and one slot call per book.

## Just want to look?

The evidence is published, and you can read it without installing anything:

- **[bk2022/world-lattice](https://huggingface.co/datasets/bk2022/world-lattice)** — the artifact
  this package ships, with its parity audits and the two gaps it still has.
- **[bk2022/corpus-425](https://huggingface.co/datasets/bk2022/corpus-425)** — 425 books slotted
  independently by three engines, 10,634 placements, and the test records around them.

Source repo: [bk2022-ui/lattice](https://github.com/bk2022-ui/lattice).

## Licence

Code: **Apache-2.0**. The artifact (`world-lattice.json`) and the corpus: **CC-BY-4.0**.

Attribution is the point. If the lattice is doing work for you, cite it.
