Metadata-Version: 2.4
Name: ndos
Version: 0.1.0
Summary: Recover, describe and query animal neuroscience data you did not organise yourself
Author: Elnaz Alikarami
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/Elnazkarami/N-DOS-
Project-URL: Repository, https://github.com/Elnazkarami/N-DOS-
Keywords: neuroscience,electrophysiology,data-management,provenance,BIDS,NWB,research-data
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Provides-Extra: test
Requires-Dist: jsonschema>=4; extra == "test"
Dynamic: license-file

# NDOS: Neuroscience Data Organization System

**Make sense of animal neuroscience data you did not organise yourself.**

A drive arrives. The person who filled it has graduated. Somewhere in it are
three years of recordings, and nobody left can say which animal is which, what
was injected when, or which of four folders called `final` is the one the
figures came from.

NDOS reads that drive without changing it, and tells you what is on it — what
it observed, what it inferred, and what it could not determine. It reconstructs
the experiment behind the files: animals, surgeries, sessions, acquisitions and
the analyses built on them. Then it lets you ask scientific questions of the
result and see the evidence behind every answer.

Three things it does that a file browser cannot:

- **Recovery.** Inventories heterogeneous storage, reads inside archives
  without extracting them, and proposes a structure — showing the rule and the
  evidence behind every guess, so you can correct it rather than trust it.
- **Discovery.** Builds cohorts from incomplete records and returns three
  answers, not two: matched, excluded, and **cannot be ruled out**. A session
  whose species nobody wrote down is not a session known not to be a mouse, and
  a tool that conflates those hands you a biased cohort without saying so.
- **Traceability.** Records what produced a result, and traces a figure back
  through the commands and parameters to the raw files it came from.

Nothing moves or changes unless you ask. Four commands write; each shows a plan
and waits for confirmation.

**The layout** is one part of this, not the whole of it.
[SPECIFICATION.md](https://github.com/Elnazkarami/N-DOS/blob/main/SPECIFICATION.md)
states a directory structure, session shape, identifiers and data flags that a
project can be checked against, and `ndos validate` reports whether one
conforms. It is a practical convention rather than a rival to BIDS or NWB —
NDOS hands off to both — and it is deliberately separable from everything
above, so a lab that has already chosen a layout can still use the rest.

> **Status:** 0.1, and honest about it. Everything documented below works and
> is tested against real lab storage on Linux, macOS and Windows. The
> specification is a draft. Interfaces may change. No external lab has
> completed a pilot yet — [that is what we are looking for](#looking-for-pilot-testers).

## Looking for pilot testers

If you have a directory of lab data nobody fully understands any more, that is
exactly what this needs to meet. It takes about fifteen minutes and **will not
move or change your data**.

| | |
| --- | --- |
| **[Tell us how it went →](https://github.com/Elnazkarami/N-DOS/issues/new?template=3-pilot-feedback.yml)** | Even "I gave up at step three" is a result |
| **[It read my folders wrongly →](https://github.com/Elnazkarami/N-DOS/issues/new?template=1-layout-misread.yml)** | The most useful report there is |
| **[Something is broken →](https://github.com/Elnazkarami/N-DOS/issues/new?template=2-bug.yml)** | A crash or a wrong answer |

No data needs sharing — every form asks about the *shape* of your directories,
never their contents. Start at **[QUICKSTART.md](https://github.com/Elnazkarami/N-DOS/blob/main/QUICKSTART.md)**.

---

## Install nothing

NDOS Core is **Python 3.9+ standard library only**. No `pip install`, no
environment, no dependencies. This is deliberate: the machine that most needs
to be inventoried is often an acquisition PC where you are not allowed to
install anything.

```bash
git clone https://github.com/Elnazkarami/N-DOS-.git ndos
cd ndos
python3 ndos.py --help
python3 ndos.py report /path/to/your/data
```

Every module is also a standalone script — `python3 ndos_report.py ...` works
identically, and a single file can be copied out and run on its own.

If you would rather type `ndos report` than `python3 ndos.py report`:

```bash
pip install ndos          # once published
pip install -e .          # or from a clone
```

Either adds the command and nothing else — there are no dependencies to
install, and CI fails the build if that ever stops being true. **New here? Start with [QUICKSTART.md](https://github.com/Elnazkarami/N-DOS/blob/main/QUICKSTART.md).** When NDOS reads your
data wrongly — and on some layout it will — [RECIPES.md](https://github.com/Elnazkarami/N-DOS/blob/main/RECIPES.md) is what
to do about it.

---

## Safety guarantee

`ndos_scan.py` and `ndos_report.py` **never modify, move, rename, extract, or
delete anything** below the directory you point them at. They open files only
to read bytes for checksums. This is verified by tests that snapshot every
size and modification time before and after a scan.

Nothing in NDOS Core writes to your data without an explicit, reviewable plan
that you approve first.

---

## The whole thing, end to end

Starting from a directory nobody understands:

```bash
python3 ndos.py report   /path/to/chaos                     # what is in here?
python3 ndos.py archive  inspect /path/to/chaos -c arch.json    # what is in the zips?
python3 ndos.py organize apply /path/to/chaos -d ./project  # build the N-DOS layout
python3 ndos.py table    export ./project -d ./metadata     # fill in what only you know
python3 ndos.py table    check  ./metadata --emit linked.json
python3 ndos.py query    linked.json -w species=mouse -w target_region=CA1
python3 ndos.py convert  bids ./project -d ./bids-export --write
```

Nothing in that sequence moves or modifies your data. The layout is built
from symbolic links; the only commands that ever write to your files are
`ndos_organize --mode move`, `ndos_archive extract`, and `ndos_tags sweep
--apply`, each of which shows a plan and asks first.

## Modules

The order below is the order you meet them: understand what you have,
then structure it, then describe it, then use it.

### `ndos_report.py` — understand a directory you have inherited

Point it at a folder and get a readable account of what is in there, what is
duplicated, how it appears to be organised, and what needs attention. Requires
no metadata and no prior setup.

```bash
python3 ndos_report.py /path/to/data                        # readable summary
python3 ndos_report.py /path/to/data -f markdown -o report.md
python3 ndos_report.py manifest.json -f json                 # machine-readable
```

It reports:

| Section | What it answers |
| --- | --- |
| What is in here | Composition by category and size — ephys, imaging, behaviour, analysis, archives |
| Inferred folder structure | Which directory level looks like a subject, a session, a date — and which names contradict that |
| Duplicate files | Byte-identical copies and how much space they waste |
| Largest directories | Where the volume actually lives |
| Needs attention | Unextracted archives, zero-byte files, unreadable paths, names that break on other systems |

Compound formats are recognised where a bare extension is not enough:
`M01_g0_t0.imec0.ap.bin` is reported as electrophysiology, not as an anonymous
`.bin`. Genuinely ambiguous extensions are labelled *ambiguous* rather than
guessed at — a confident wrong label is worse than an honest unknown.

Inferred structure is always labelled as inference. NDOS distinguishes what it
**observed**, what it **computed**, and what it **guessed**, and never presents
one as another.

### `ndos_scan.py` — read-only inventory

Produces a versioned JSON manifest: every file with its path, size,
modification time, and SHA-256 digest, plus an explicit list of everything
skipped and why.

```bash
python3 ndos_scan.py /path/to/data --output manifest.json
python3 ndos_scan.py /path/to/data --no-checksum        # faster, no dedup
python3 ndos_scan.py /path/to/data --exclude '*.tmp'
```

The manifest is the input to every other NDOS module, so a slow checksummed
scan only has to happen once. The output contract is versioned in
[`schemas/manifest.schema.json`](https://github.com/Elnazkarami/N-DOS/blob/main/schemas/manifest.schema.json).

An inventory that quietly omits data is worse than no inventory, so
unreadable directories, permission failures, and symlinks are recorded in a
`skipped` list rather than silently dropped.

### `ndos_archive.py` — see inside archives without extracting them

Labs zip their archives because the data is enormous. On a real lab drive,
**88% to 100% of everything was inside `.zip` files**, which meant no
inventory could describe any of it.

This reads an archive's index rather than its contents, so a 300 GB collection
can be catalogued without unpacking a byte:

```bash
python3 ndos_archive.py inspect /Volumes/archive -c archives.json
python3 ndos_archive.py search archives.json '*.avi'
```

Listings are cached and keyed on size and modification time. That matters: on
a slow external drive the first read of a 2 GB archive took nearly a minute,
and nobody should pay that twice.

**Extraction is planned, then confirmed, then done.** Ask for what you want,
see exactly what it would write and what it would cost, and only then say yes:

```bash
python3 ndos_archive.py plan archives.json --dest ./work --name '*.avi'
python3 ndos_archive.py extract archives.json --dest ./work --name '*.avi'
```

The plan reports how many files, how many bytes, which already exist, and
whether there is enough free space — refusing to start if there is not.
Nothing is written until you confirm.

**Members that would escape the destination are refused.** An archive can
contain `../../etc/something`, and unpacking one you did not create is a real
way to get files written where you did not intend. Those members are listed
under REFUSED and never extracted.

`.tar` and `.tar.gz` need `--include-tar`, because unlike a zip they must be
streamed end to end to be listed, which on slow storage is expensive enough to
be a deliberate choice.

### `ndos_organize.py` — rebuild the N-DOS layout from what already exists

An inventory *describes* a directory. It does not make an inherited archive
understandable to a PI whose data collector left years ago. This builds the
missing half: the N-DOS project, derived from the original structure rather
than imposed on it.

```bash
python3 ndos_organize.py plan  /path/to/chaos -d ./project   # see it first
python3 ndos_organize.py apply /path/to/chaos -d ./project   # then confirm
python3 ndos_organize.py undo  ./project/.ndos-layout-log.json
```

The layout is the one defined in the N-DOS manuscript:

```
project/
├── raw_data/<SubjectID>/<SessionID>/     # acquisition files
├── processed_data/  analysis/  derivatives/
├── flagged_data/    figures/   scripts/   metadata/
└── README.md
```

SessionID follows the manuscript's `YYYYMMDD` convention, becoming
`YYYYMMDD_01`, `_02` when a subject was recorded more than once that day.

**Nothing is copied or moved by default.** The tree is built from symbolic
links, so a 45 GB collection is organised in seconds, occupies about 40 KB,
and is undone by deleting it. `--mode copy` and `--mode move` exist, are
planned and confirmed the same way, and `move` says plainly that it relocates
your data. Every apply writes `.ndos-layout-log.json`, and `undo` reverses it —
including moving files back.

**Filenames follow the manuscript's conventions**, `SubjectID_SessionID_type`:

```
A0634_20201122_video-0.avi        A0634_20201122_raw-info.rhd
A0634_20201122_position-Take-2020-11-22-06.32.30-PM.csv
A0634_20201122_experimenter-notes.csv
```

The data type comes from the file itself — its extension, then its name — and
never from the directories above it, which decide the *role* instead. Where no
standard type applies, the original descriptor is kept (`_analogin.dat`)
rather than forcing a file into a category it may not belong to: a confident
wrong label is worse than an unfamiliar one, because the filename is what
everyone reads first. Pass `--keep-original-names` to skip renaming entirely.

Because the default mode is links, renaming costs nothing and risks nothing:
the link carries the standard name while the file it points at keeps its own.

**Sessions are clustered, not split.** A miniscope starting at 18:32:25 and an
Intan at 18:32:20 are one recording, not two. Acquisition times within ten
minutes are treated as one session; genuinely separate recordings on a day
become `YYYYMMDD_01` and `_02`.

**Every placement explains itself:**

```
raw_data/A0634/20201122/0.avi
    from  0.avi
    why   folder 'A0634' matches a subject identifier;
          'A0600' above it read as a cohort or range
    why   date '2020_11_22' with acquisition time '18_32_25'
    why   imaging and video is treated as raw_data
```

Structure is read from directory names, from filenames when the folders are
silent (`2020_11_20.zip` carries its date nowhere else), and from compound
names like `A0634_201122_183220`.

Where the folders record a session but not when it happened, `--dates` takes
the date from your `sessions.csv`, turning `ses-01` into `20250314`. A dated
identifier is what the standard establishes, because it places a recording in
time without opening anything. A date already in the path is never
overridden.

**Analysis-tool output is left alone.** A Phy or Kilosort sorting is opened by
looking for `spike_times.npy` and `params.py` by name, so renaming inside one
would stop the tool reading it back. Directories written by Phy, Kilosort,
SpikeInterface, suite2p, Open Ephys, DeepLabCut or Zarr are recognised by the
files those tools require, placed under the session they belong to, and carried
across with their filenames and internal structure untouched. The plan says
when it has done this and why.

**Nothing is dropped.** Files whose subject or session cannot be determined go
to `flagged_data/` with their original structure intact and a
`flagged_notes.json` saying why — which is what the N-DOS layout reserves that
directory for.

**Redundant copies are recognised, not duplicated.** A lab that restructured
its data once has the same recording in two places; on a real drive this was
3.0 GB. Each file is linked once and the copies are listed, rather than
appearing under invented names as though they were distinct.

### `ndos_table.py` — get lab metadata in, via the tool you already use

Scanning reveals what is on disk. It cannot reveal which animal a recording
came from, what was injected, or when. That knowledge lives in a notebook or
an Excel sheet, so NDOS meets it there.

```bash
python3 ndos_table.py export manifest.json -d metadata/   # build the sheets
# ... open them in Excel and fill in the blanks ...
python3 ndos_table.py check metadata/ --emit linked.json
```

Metadata lives in **three linked tables**, because a fact recorded once should
govern every session it applies to:

| File | One row per | Filled by |
| --- | --- | --- |
| `animals.csv` | animal — species, strain, sex, date of birth, genotype | you, once per animal |
| `procedures.csv` | surgery, injection, implant, drug, training | you; NDOS cannot observe a surgery, so it never touches this file |
| `sessions.csv` | recording session — date, task, QC | NDOS pre-fills what it observed; you add the rest |

`animals.csv` is seeded with the subject names NDOS found in your folder tree,
so you start with rows rather than a blank sheet. Sessions are regenerated on
every export with observed columns refreshed and typed-in values carried
across; a row that disappears is reported rather than silently taking its
metadata with it.

**Intervals are computed, never typed.** Because a procedure has a date and a
session has a date, NDOS derives `days_since_injection`, `days_since_implant`,
`age_days`, and so on. These carry the status `computed`, so they are never
mistaken for something a person asserted — and they make "recorded three to
five weeks after the injection" a query you can actually run.

Entry is forgiving, validation is strict. `mouse`, `Mouse`, and `mice` all
resolve to `mus musculus`; `ephys` to `electrophysiology`; `viral injection`
to `injection`; `female` to `F`. What you typed is preserved beside the mapped
value so the mapping can be audited. But `21/03/2025` is refused, because
`03/04/2025` means 3 April in the UK and 4 March in the US and guessing would
silently corrupt a date.

Validation spans the tables, not just each file: a session naming an animal
with no row, or a procedure for a subject nobody described, is reported as a
broken link.

A blank cell and the word `unknown` mean different things, and NDOS keeps them
apart: blank means nobody has filled it in yet, `unknown` means somebody
checked and could not determine it. `--emit` writes evidence-typed records
against [`schemas/session_metadata.schema.json`](https://github.com/Elnazkarami/N-DOS/blob/main/schemas/session_metadata.schema.json).

### `ndos_tags.py` — validated, temporary, safe to delete

The layout reserves `flagged_data/` and `temp/`, and the standard defines the
flags that make them mean something:
`{"validated": true, "temp": false, "deletable": false}`.

```bash
python3 ndos_tags.py set spikes.npy --validated --note "curated in Phy"
python3 ndos_tags.py list ./project --flag temp
python3 ndos_tags.py sweep ./project              # plan a cleanup
python3 ndos_tags.py sweep ./project --apply      # after reading it
```

Tags live in a `tags.json` beside the data they describe, one per session, so
a session directory stays self-describing if it is moved or copied.

**A validated file is never swept**, whatever its other flags say, and marking
something both validated and deletable is recorded as a conflict rather than
resolved silently. `sweep` plans by default and deletes only on `--apply` with
a confirmation: the manuscript imagines a maintenance script removing
temporaries automatically, and deletion driven by a hand-edited flag is how a
lab loses data it meant to keep.

Files that *look* like scratch but were never flagged are listed separately
and only with `--include-untagged`, because nobody has vouched for them.
Scratch is judged relative to the project root, so a project living under
`/tmp` does not have all of its files called temporary.

`ndos_organize` tags automatically: spike-sorting scratch is routed to
`processed_data/<sub>/<ses>/temp/`, flagged `temp`, and recorded in that
session's `derived_metadata.json` — so a sweep finds it later without anyone
remembering which files were intermediates.

### `ndos_validate.py` — does this project follow the standard?

```bash
python3 ndos.py validate ./my-study
```

```
  This project does not yet conform.

REQUIRED (2)
  session 'March14' is not YYYYMMDD or YYYYMMDD_NN
      at    raw_data/M123/March14
      fix   rename it, or rebuild the layout with ndos organize
```

Requirements and recommendations are kept apart. A requirement unmet means the
project does not conform and the command exits non-zero, so it can gate a
hand-off or a submission. A recommendation — raw data still writable, files not
following the naming convention, no metadata yet — is reported and never
affects the exit code, because a lab may depart from those deliberately.

Every finding says what to run to fix it.

### `ndos_protect.py` — raw data read-only after acquisition

The standard asks that raw data be set read-only once acquired. A recording is
the one thing in a project that cannot be regenerated, and it is usually lost
not to a disk failure but to a script writing where it meant to read.

```bash
python3 ndos.py protect ./my-study            # what it would change
python3 ndos.py protect ./my-study --apply
python3 ndos.py protect ./my-study --check    # has anything become writable?
python3 ndos.py protect ./my-study --release
```

Only `raw_data/` by default; `processed_data/` is meant to change. Permissions
are changed and contents never are. `--check` exits non-zero when something
that should be read-only is not, so a scheduled job or a pre-publication check
can use it.

Releasing is as easy as protecting, deliberately: a protection people cannot
undo is one they work around by copying data somewhere unprotected.

### `ndos_query.py` — build a cohort, and see why each session qualified

```bash
python3 ndos_query.py metadata.json -w species=mouse -w sex=F \
    -w 'target_region=CA1' -w 'session_date>=2025-03-01' \
    -w 'modalities~electrophysiology' \
    --save-cohort cohort.json --name ca1-ephys-spring-2025
```

Operators are `=` `!=` `>` `<` `>=` `<=` and `~` (contains). `field=*` requires
any value; `field=?` finds values recorded as unknown. Queries are normalised
the same way the data was, so `species=mouse` finds sessions recorded as
`mus musculus`, and the report shows you that substitution.

**Results come in three groups, not two.** A session whose sex was never
recorded is not a non-match — it is an open question, and quietly dropping it
biases the cohort in a way nobody sees. So NDOS reports:

| Group | Meaning |
| --- | --- |
| Matched | Every criterion satisfied, each citing the field, value, and evidence status it rests on |
| **Cannot be ruled out** | Met everything checkable, but a deciding value was never recorded |
| Excluded | Ruled out by evidence that *is* recorded |

Unresolved sessions stay out of a saved cohort unless you pass
`--include-unresolved`, and are labelled if you do.

**An empty result explains itself** rather than returning nothing. It names the
constraint that eliminated the most sessions, and distinguishes "your query was
too narrow" from "this column was never filled in" — which need opposite fixes.

Cohorts are frozen with the full query plan against
[`schemas/cohort.schema.json`](https://github.com/Elnazkarami/N-DOS/blob/main/schemas/cohort.schema.json), including counts of
what was excluded and what could not be decided, so a selection can be re-run,
audited, or disputed later.

### Declaring which data suits which analysis

```bash
python3 ndos_query.py linked.json --config analyses.json
```

```json
{"analyses": [
  {"name": "theta-power-CA1",
   "requires": ["target_region=CA1", "days_since_injection>=21", "qc_status=pass"]}
]}
```

Each analysis is a saved query, so the answer carries the same evidence rules:
sessions that qualify, sessions that cannot be ruled out, and why nothing
matched when nothing does.

### `ndos_prov.py` — record what produced a result

Provenance normally goes uncaptured because capturing it means instrumenting
analysis code, and nobody rewrites a working script to satisfy a data policy.
So NDOS wraps the command instead:

```bash
python3 ndos_prov.py run --input raw/ --output processed/ \
    -- python3 preprocess.py raw processed
```

Nothing about `preprocess.py` changes. NDOS checksums the inputs, watches the
output directories before and after, records the git commit and environment,
and writes a run record. The wrapped command's exit code passes through, so
this composes inside existing shell scripts.

Then walk backwards from any result to the data behind it:

```bash
python3 ndos_prov.py trace figures/figure1.svg
```

```
figure1.svg
└── → figure  [run-157f69ae502a]  2026-08-20T02:39:50Z
      $ python3 scripts/make_figure.py processed figures
    ├── M01_ses01_filtered.dat
    │   └── → preprocess  [run-418c7c94596d]  2026-08-20T02:39:49Z
    │         $ python3 scripts/preprocess.py raw processed
    │       ├── M01_ses01.dat
    │       │   [raw] not produced by any recorded run
```

**Failed runs are recorded too**, and marked. "This figure came from a script
that exited 1" is exactly what someone needs warning about, so a partial
output is captured rather than discarded.

**Provenance never collects credentials.** Environment variables are recorded
only when named explicitly with `--record-env`; capturing the whole
environment would routinely bury API keys inside files meant to be shared.
`--anonymous` additionally omits hostname and username, for provenance you
intend to publish.

Records validate against
[`schemas/provenance.schema.json`](https://github.com/Elnazkarami/N-DOS/blob/main/schemas/provenance.schema.json) and follow
the W3C PROV shape of an activity that *used* and *generated* artifacts.

### `ndos_convert.py` — hand off to BIDS and NWB

NDOS does not reimplement either standard. NWB conversion is solved by
maintained tools, and rewriting it here would produce a worse converter nobody
maintains. This prepares the handoff instead:

```bash
python3 ndos_convert.py bids ./project -d ./bids-export --write
python3 ndos_convert.py nwb  ./project --metadata linked.json --save nwb-plan.json
```

`bids` builds a BIDS-shaped tree of links with `dataset_description.json`,
`participants.tsv` and per-file sidecars, mapping N-DOS entities onto BIDS
ones — `A0634` → `sub-A0634`, `20201122` → `ses-20201122`, and the
discriminator onto `acq-`. Scratch in `temp/` is never published.

**It is BIDS-shaped, not validated BIDS**, and says so in its own output.
Animal electrophysiology is covered by BEP032, which is not finalised, so no
export can honestly claim conformance today.

`nwb` emits a conversion plan with metadata already mapped onto NWB's fields,
ready for NeuroConv or a lab script — and reports what is still missing
(`species`, `sex`, `date_of_birth`) rather than inventing it.

### `ndos_init.py` — start an NDOS project

Creates a project profile and the directories NDOS owns. It does **not**
migrate, move, or restructure your data; raw data stays exactly where it is.

```bash
python3 ndos_init.py ~/projects/my-study
```

---

## `ndos gui` — the same tools, in a window

Not everyone in a lab works at a command line, and the person who knows what
the data *is* often is not the person who is comfortable there.

```bash
python3 ndos.py gui        # or: ndos gui
```

That serves a local page and opens it. Four things it does, each calling the
same functions the commands call rather than a second implementation that
could disagree with them:

| | |
| --- | --- |
| **This machine** | Pick a folder, see how long a scan would take, watch it run, read the result, check it against the standard |
| **Manifest** | Open a `manifest.json` by name and read what a scan found |
| **Query** | Build a cohort, and see all three answers — matched, excluded, and **cannot be ruled out** |
| **Validate** | Check a project's metadata tables against each other, and enter the facts only a person knows |

The query page is the one worth understanding. A session that never recorded a
species is not a session known not to be a mouse, and a page that showed you
only "matched" and "everything else" would let you publish a cohort that is
quietly biased towards the animals somebody happened to write down. So the
query runs where the rules live, and answers with all three groups, the
constraint that blocked each session, and what filling it in would change.

Nothing the page does writes to your data. Linking a project's metadata for a
query builds the records in memory — `ndos table check --emit` writes them to
a file, and the page needs no such file to exist.

It is part of the package, already built, so there is still nothing to install:
no Node, no npm, no build step, and nothing fetched while it runs.

**It is not a website and cannot be made into one.** The server binds to the
loopback address, requires a token that only the page it opened is given,
refuses any request that does not claim a local `Host`, and refuses any request
carrying a foreign `Origin` — which is what stops a page you happen to have
open in another tab from reading your disk. Nothing is exposed to the network,
and nothing leaves the machine.

The command line remains the primary interface: everything the page can do, a
command can do, and some things only a command can do.

---

## Try it without any data

A synthetic messy lab project is included, containing problems chosen because
they recur in real labs: a duplicated backup, an unextracted archive, a failed
acquisition left as a zero-byte file, inconsistent subject naming, and
histology stored away from the recordings it validates.

```bash
python3 tests/fixtures/make_messy_lab.py /tmp/messy-lab
python3 ndos_report.py /tmp/messy-lab
python3 ndos_table.py export /tmp/messy-lab -d /tmp/metadata
# fill in a few rows across the three sheets, then:
python3 ndos_table.py check /tmp/metadata --emit /tmp/linked.json
python3 ndos_query.py /tmp/linked.json -w species=mouse -w sex=F
```

---

## Tests

```bash
python3 -m unittest discover -s tests -v
```

The suite runs on the standard library alone. If `jsonschema` is installed, it
additionally validates generated manifests against the published schema.

---

## Repository layout

| Path | Contents |
| --- | --- |
| `ndos.py` | One command dispatching to all of the below |
| `QUICKSTART.md` | Fifteen minutes, for a lab member |
| `RECIPES.md` | What to do when a guess is wrong |
| `ndos_report.py` | Inventory report |
| `ndos_scan.py` | Read-only inventory |
| `ndos_archive.py` | Archive inspection and planned extraction |
| `ndos_organize.py` | Rebuild the N-DOS layout from existing structure |
| `ndos_table.py` | Linked metadata tables |
| `ndos_tags.py` | Validation flags, cleanup, validated-file index |
| `ndos_protect.py` | Make raw data read-only after acquisition |
| `ndos_query.py` | Cohort queries with evidence citation |
| `ndos_prov.py` | Run provenance and lineage tracing |
| `ndos_convert.py` | BIDS and NWB handoff |
| `ndos_init.py` | Start a project, and make a session folder |
| `ndos_validate.py` | Check a project against the standard |
| `ndos_gui.py` | Serves the local interface, on this machine only |
| `ndos_gui_static/` | That interface, already built, so no Node is needed |
| `gui/` | Its source, for changing it — see [gui/README.md](https://github.com/Elnazkarami/N-DOS/blob/main/gui/README.md) |
| `scripts/` | Regenerating what is derived: field definitions, the built interface |
| `SPECIFICATION.md` | **The standard itself** |
| `schemas/` | Versioned JSON Schema contracts |
| `tests/` | Test suite and synthetic fixtures |
| `LEGACY.md` | Where the superseded prototypes went, and why |

Scripts that predate NDOS Core are no longer in this branch: they move data
with no rollback and one of them contradicts the standard. They remain
reachable, with their history — see [LEGACY.md](https://github.com/Elnazkarami/N-DOS/blob/main/LEGACY.md).

---

## Scope

**In scope:** non-human animal neuroscience — colony and cohort context,
surgeries, injections, implants and drugs, behavioural training, extracellular
electrophysiology and calcium imaging, tissue collection and histology, and the
analyses derived from them.

**Not in scope:** replacing BIDS for human MRI/MEG/EEG, replacing NWB as a
container, or becoming a public archive. NDOS reads and hands off to those
ecosystems rather than competing with them.

---

## Licence

[Apache-2.0](LICENSE). Chosen over MIT for its explicit patent grant, which
institutional legal review generally looks for before adoption.

Earlier revisions of this repository declared MIT in the README. That grant
still stands for those revisions; anyone who took the code under MIT keeps it.
NDOS Core is Apache-2.0 going forward.
