Metadata-Version: 2.5
Name: ingence
Version: 0.3.0
Summary: Ingence is a read-only indexing and inference engine and local Dashboard for exploring scientific filesystems: it surveys what is there, and proposes how files relate using a Dictionary and a Rulebook you can read and edit.
Project-URL: Homepage, https://ingence.org/
Project-URL: Documentation, https://ingence.org/
Author: Andrew Bazley
License-Expression: MIT
License-File: LICENSE
Keywords: filesystem,inference,metadata,provenance,research-data-management
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: MacOS
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.12
Requires-Dist: blake3>=0.4
Requires-Dist: duckdb>=1.0
Requires-Dist: numpy>=1.26
Requires-Dist: olefile>=0.47
Requires-Dist: openpyxl>=3.1
Requires-Dist: pyarrow>=16
Requires-Dist: pydantic>=2.7
Requires-Dist: pypdf>=4.2
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.7
Requires-Dist: scikit-learn>=1.5
Requires-Dist: scipy>=1.13
Requires-Dist: typer>=0.12
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == 'api'
Requires-Dist: uvicorn>=0.29; extra == 'api'
Provides-Extra: browser
Requires-Dist: playwright>=1.55; extra == 'browser'
Provides-Extra: cluster
Provides-Extra: content
Requires-Dist: datasketch>=1.6; extra == 'content'
Requires-Dist: nbformat>=5.10; extra == 'content'
Requires-Dist: pillow>=10.3; extra == 'content'
Requires-Dist: pyahocorasick>=2.0; extra == 'content'
Requires-Dist: tifffile>=2024.1.30; extra == 'content'
Provides-Extra: detect
Requires-Dist: magika>=0.6; extra == 'detect'
Requires-Dist: python-magic>=0.4; extra == 'detect'
Provides-Extra: dev
Requires-Dist: hypothesis>=6.100; extra == 'dev'
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pre-commit>=3.7; extra == 'dev'
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0; extra == 'dev'
Provides-Extra: docs
Requires-Dist: myst-parser>=5.1; extra == 'docs'
Requires-Dist: pydata-sphinx-theme>=0.23; extra == 'docs'
Requires-Dist: sphinx-design>=0.7; extra == 'docs'
Requires-Dist: sphinx>=9.1; extra == 'docs'
Provides-Extra: images
Requires-Dist: imagehash>=4.3; extra == 'images'
Requires-Dist: pylibczirw>=4.1; extra == 'images'
Provides-Extra: patterns
Provides-Extra: themes
Description-Content-Type: text/markdown

# Ingence

Ingence is a read-only indexing and inference engine for scientific filesystems. It reads files’ names, metadata and contents where they already sit, and interprets them through two editable, versioned sources: a Dictionary, which defines what the words in names and files mean, and a Rulebook, the engine’s interpretive logic, which turns that evidence into proposals about how files relate. Every conclusion names the rule and evidence behind it, and where the rules cannot decide, the engine abstains.

The Dashboard, served locally in your browser, is where you explore the results: search for a file or experiment, see what was read from it and what was not, follow how files connect, and answer the questions the engine could not settle. A survey of what a filesystem holds is useful on its own, before any inference is confirmed.

## Maturity: pre-alpha

Ingence is a pre-alpha. It runs from start to finish and is changed often; what it produces are
proposals and readings, not scientific conclusions.

<!-- render:begin feedback -->
<!-- Made from release.toml by tools/release/render.py; edit release.toml, not these lines. -->
Send feedback to hello@ingence.org. Say what you ran and what you expected.
Before you send anything, see [Reporting a problem without sharing your data](https://ingence.org/guides/reporting-a-problem.html).
<!-- render:end -->

## The workflow

`ingence see`, with no project, opens a page in your browser. The page makes a project, asks the
few things the engine needs before a first run, and runs the steps below when you click, showing
what it has seen so far (`GETTING_STARTED.md`). The same steps by hand:

```
ingence init PROJECT --root /path/to/data     # declare what to read; writes only PROJECT
ingence index -p PROJECT                      # the census: every file's metadata, no file opened
ingence dive  -p PROJECT --budget 900         # open files within a time budget
ingence infer -p PROJECT                      # evaluate the rules in force and publish
ingence see   -p PROJECT                      # the Dashboard, on this machine only
```

- **`init`** creates a project directory wherever you choose (see *Where a project lives*). A root
  is a folder, or a public archive's listing (`listing:<archive>:<identifier>`, fetched once with
  `index --fetch`). For someone else's folder, `init --depth metadata` declares once that no dive
  there opens a file.
- **`index`** walks each root and records what the filesystem states (names, sizes, times,
  links) without opening a file, then reads every name into its parts against the Dictionary.
  `--rebind` reads every stored name again under the current vocabulary and decisions without
  touching the root; `--dry-run` reports what a census would find and writes nothing.
- **`dive`** opens files within a time budget, one file type at a time. It reads headers,
  instrument clocks, text, tables and archives' member lists, in supervised subprocesses.
- **`infer`** evaluates the Rulebook over the current run. Each conclusion is a claim that carries
  its rule, the rule's version and its evidence. A rule that cannot decide abstains and says why.
  A rule that proposes a pair or a group asks a person and does not assert it.
- **`see`** serves the Dashboard on the loopback interface (an address that only your own
  computer can reach). Its pages are Explore (search and inspect), Graph, Questions, Engine and
  History. An answer is written to the Record, the append-only log of human decisions, and waits
  for Re-run with every other change. Re-run reads the affected names again and runs the rules
  again, and until then the pages keep showing the last published run.

`rule`, `teach`, `record`, `log`, `export`, `map` and `fix` read and revise the Rulebook and the
Dictionary, import review sheets, show the Change Log, and diagnose a project.

`index`, `dive` and `infer` each end by saying what was read, what waits for you and the next
command; `--verbose` adds the detail (each rule's counts, reuse, timings).

## Domain packs and proposals

The core Dictionary reads what every laboratory holds: file formats, dates, units, stages. A
**domain pack** adds the words and file types of one science (`microscopy`, `biochemistry`,
`sequencing`, `spectrometry`, `electrophysiology`, `clinical`, `computational`). A pack adds
vocabulary only, with no rules of its own. A new project loads the packs that `init --domain`
names (repeat it for several), or every pack with `--all-domains`. With neither, it loads the core
alone. `programs`, which is about software and not a science, always loads. Every pack ships with
the engine. Where the core and a pack spell a word alike, a person's answer wins. Otherwise the
core's meaning is used over a shipped pack's, unless the project has accepted that pack. The
Dictionary page lists those words.

**A pack of your own** is one YAML file. `ingence teach export-answers --out ours.yaml` writes
what your project was taught, and `ingence teach pack install <file, folder or https://address>`
installs a pack. The pack's `pack:` block gives its name, version, author, licence, the schema it
needs and a digest of its words. Ingence refuses a pack whose words do not match its digest, or
that needs a newer Ingence. Your own pack stands level with the core, so a word both hold waits for
your answer. Once the project accepts a shipped pack, the packs it chose, yours among them, stand
above the core, and the other shipped packs stand level with it. A pack can be kept loaded
without standing over the core. Every word of an installed pack is active.

**The engine proposes; a person decides.** After `infer`, the Dashboard's Questions page asks
about:

- **A pack to load**, where a pack the project does not load would read enough of what nothing
  reads, or where three or more files are in a format its instruments write (ND2 for microscopy).
  The question shows its counts. Accepting loads the pack from the next run. Refusing stops the
  question.
- **A pack to keep**, in a project that loads every pack, where no name holds one of its words.
  Refusing stops loading the pack from the next run.
- **Groups of experiments** that share words, most general first (on Groupings).
- **Pairs of files** where one may have been made from the other (on Proposed pairs).
- **A word nothing reads**, where words of one kind stand, and what it means.

Nothing is loaded, grouped or derived until someone says yes.

## Links on a share

A link is recorded and never followed into. By default (`scan.resolve_symlinks: true` in the
project's `settings.yaml`) the census asks whether each link's target is there, so it can tell a
link whose target is gone from one whose target is elsewhere. It follows a chain one link at a
time, and only inside the root. The first link whose text points outside the root is read from
that text and never followed, whatever the setting. On a slow share each of those questions waits
on the share. Turn them off with

```
scan:
  resolve_symlinks: false
```

and each link is then recorded as a link, with its target unasked.

## After `see`: the Dashboard

`ingence see -p PROJECT` opens the Dashboard in your browser. It has five pages:

- **Explore**: what is here, what could not be read, and a search box for an experiment or a
  file. Open any file to see what was read from it and every conclusion about it.
- **Graph**: how files relate (what contains what, what was made from what, what mentions
  what), from one folder or file you choose.
- **Questions**: everything waiting for you, one section per kind, each with how many are
  waiting: Questions (what no rule could decide, the most useful first), Groupings, Claim
  samples, Proposed pairs, Extensions, Programs, and Coverage (what has not been read yet, and
  why).
- **Engine**: the rules in force, each in plain words with how much of the run rests on it, the
  Dictionary, and the project's settings.
- **History**: what changed between two runs, and why.

**Every answer is written to the Record** under your name (the page asks who you are once) and
never overwrites an earlier one. **Every change waits for Re-run**: an answer, a word's meaning,
a pack, a committed setting. The count on the Re-run button says how many, and opens a summary
by kind. Nothing runs until you press it, or run `ingence index --rebind -p PROJECT` and then
`ingence infer -p PROJECT`. Opening a project where changes wait says so once, with Re-run now and
Later. While a re-run works, the pages keep showing the last published run.

**Export and Import carry the questions to a colleague.** Export saves them as one file, and
Import records each answer as its author's.

## Principles

- **Source data is read-only.** Ingence never moves, renames, deletes or writes to indexed
  files, and never executes code it finds.
- **Observations and interpretations never share a container.** A fact and an inference
  cannot be written to the same field.
- **Every interpretation is reproducible** from four inputs: the Index, the Rulebook, the
  Dictionary and the Record.
- **Abstention is a valid output.** "Unknown", "unexamined" and "conflicting" are three
  different states and stay distinct.
- **No model authors a claim.** No language model interprets your files, and none is needed
  to read Ingence's output.
- **More evidence never lowers a claim's standing.**
- **It runs on your machine.**

## Offline, and when the source is away

The engine runs entirely on the machine it is installed on. It makes network connections only
where you ask it to: installing the package, fetching a public archive's listing with
`index --fetch` (one request at a time, every request logged), and installing a pack from an
address with `teach pack install https://...`. Nothing is sent anywhere else. The Dashboard
binds `127.0.0.1` only, loads no remote script, font or image, and refuses writes that come from
other pages in your browser.

**The source can be away.** A walk (`index`) and a `dive` need the data they read, and refuse a
root they cannot reach before writing anything. Everything else reads the project directory
alone: `index --rebind`, `infer`, the Dashboard, answering questions and applying the answers
all work with a drive unplugged or a network share unmounted. The last published run stays
readable at all times: a run is published atomically or not at all, and an interrupted
command leaves the previous run current.

## Schemas and upgrades

Every record the engine writes carries a `schema_version`. `schemas/v1/` holds the record
contract in machine-readable form: one JSON Schema per record type, the eight relation
primitives with their formal properties, the specialization catalog, the publication
invariants and the storage table declarations. `schemas/api/v1/` holds the Dashboard API's
response models. Both are generated from the code.

- **A newer engine reads an older project.** A minor schema version adds fields compatibly;
  a project written by an older engine is rebound and inferred by a newer one, and the test
  suite holds projects written by earlier engines to prove it.
- **An older engine refuses a newer project** rather than misreading it. Once a newer engine
  has written into a project (a new kind of decision, a new run), an older engine stops with a
  message naming the version. Upgrade a copy first if two engines must share a project. A
  project with a field the engine does not know (a name, from schema 1.23) is refused the same
  way, saying which build made it.
- A major schema version would change a field's meaning and require a migration; there has
  been none.

## Versions

Two versions move independently: the **package version** (`ingence --version`) and the
**schema version** of the record contract it writes.

The package version is SemVer-like in its pre-1.0 form, `0.MINOR.PATCH`, written as Python's own
versions are (PEP 440), with a `.devN` suffix while pre-alpha:

- **MINOR** moves when a change makes a project unreadable by the previous package, that is,
  whenever the record schema's minor version or the Dashboard API's major version moves;
- **PATCH** moves for every other change merged to the main branch;
- **`.devN`** marks a development build between releases. A version without it is a release,
  and only a tagged commit is one.

The current version is `0.3.0`, for schema 1.23 and the versioned Dashboard API. Every run
also records the engine's commit, which identifies a development build exactly.

## Supported systems

- **Python 3.12, 3.13 and 3.14.**
- **macOS, Linux and Windows**, each tested on every change. Filesystem behavior (case
  sensitivity, links, birth times, rename semantics) is measured for each root and not assumed
  from the platform.
- **A laptop is enough**: the target machine has 12 GB of memory and four to eight cores. No
  GPU is used or required.

## Install

Ingence is on the Python Package Index. There are three ways to install it, in
[`INSTALL.md`](INSTALL.md): `uv` (the quickest), `pip` in a virtual environment, and `conda`. With `uv`:

```
uv tool install ingence
ingence --version
```

To upgrade, run `uv tool upgrade ingence`. Then read [`GETTING_STARTED.md`](GETTING_STARTED.md).
`INSTALL.md` also covers a wheel file, a checkout and the optional parts.

## Where a project lives

Anywhere: your own disk, a drive, a server, inside the folder it reads, or that folder itself.
A project inside the folder it reads adds its own files there. Every run leaves them out, with
Ingence's own installation and any other Ingence project it finds. Your own disk is best: a
project is written to on every run, and a local disk is faster and is there when a drive or
server is not.

## Licensing and credit

Source code and bundled software documentation are MIT (`LICENSE`). Synthetic datasets and
de-identified benchmark data are CC0-1.0, and the technical report is CC-BY-4.0. Both texts are
in `LICENSES/`. Third-party dependencies keep their own licenses and notices, which are listed in
`NOTICE`. Vendored web assets carry their own license beside them.

Copyright (c) 2026 Andrew Bazley. Supported by the Astera Institute through the Hidden Science
Competition.
