Metadata-Version: 2.4
Name: PyATCLI
Version: 2026.10.10
Summary: The analysis library, engine and command line behind PyAT: TDT fibre photometry, Oxysoft NIRS, Terranova EFNMR and text-field studies, with provenance records (formerly PhysicsLibrary)
Author: zakgm2
License-Expression: AGPL-3.0-only
Project-URL: Homepage, https://github.com/zakgm2/PyATCLI
Project-URL: Repository, https://github.com/zakgm2/PyATCLI
Project-URL: Changelog, https://github.com/zakgm2/PyATCLI/blob/main/CHANGELOG.md
Keywords: pyat,physics,neuroscience,fibre-photometry,NIRS,TDT,EFNMR,signal-processing,text-analysis,nlp,reproducibility
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.0
Requires-Dist: scipy
Requires-Dist: tdt
Requires-Dist: pandas
Requires-Dist: openpyxl
Requires-Dist: sentence-transformers
Requires-Dist: statsmodels
Requires-Dist: scikit-learn
Requires-Dist: tqdm
Dynamic: license-file

# PyATCLI

The analysis library, engine and command line behind [PyAT (Python Analysis Tool)](https://github.com/zakgm2/PyAT) — file parsing, signal processing, curve fitting, group statistics, saved state and provenance records, with no GUI and no plotting code of its own: it computes and writes numbers, and PyAT draws them. Any interface (a script, a notebook, `python -m PyATCLI`) can sit on top of it.

> **Coming from PhysicsLibrary?** This is the same project under a new name, from version 2026.10.10 (PyPI: `ZaksPhysicsLibrary` → `PyATCLI`). Change `import PhysicsLibrary` to `import PyATCLI`, `python -m PhysicsLibrary` to `python -m PyATCLI` and the `physicslib` command to `pyatcli`; nothing else about the top-level names changed. [CHANGELOG.md](CHANGELOG.md) has the list, including the modules that moved into folders.

---

## What it does

- **Loads lab data** from three instrument formats plus generic tabular files:
  - **TDT** (Tucker-Davis Technologies) fibre photometry tanks
  - **Oxysoft / Artinis** (Oxymon, OctaMon, PortaMon …) NIRS `.txt` exports
  - **Terranova Prospa** `.pt2` EFNMR/MRI 2D images
  - Generic **Excel / CSV / TSV / plain text**, with automatic sub-table detection for side-by-side data layouts on one sheet
- **Processes signals** — RANSAC-robust motion correction, bleach correction, denoising, Z-score PETH slicing, FFT with peak annotation, slope/segment analysis
- **Fits curves** — linear, single/double exponential, exponential rise, Gaussian, sinusoidal, and a photon-entanglement visibility model, all via `scipy.optimize.curve_fit`
- **Analyses text field studies** — one JSON file per subject with several free-text fields; pick any pair of fields to compare directly (e.g. does the answer to one question track another for the same subject). Word counts, data-quality flagging, sentence-transformers embeddings, an optional delta-vector magnitude between two fields, and an optional paired-similarity metric per pair with a permutation test and a word-count confound check. Domain-agnostic — field names and which pairs to compare are supplied by the caller, nothing is hardcoded to one study
- **Validates the similarity metric statistically** — Benjamini-Hochberg FDR-corrected permutation-test p-values, Cohen's d effect size, a word-count-controlled OLS regression (statsmodels), a bootstrap confidence interval on the mean, and a leave-one-out sensitivity check, one row per field pair, with docstrings explaining what each statistic means
- **Opens a recording the way PyAT does, without the GUI** — a TDT block processed to dF/F with its saved splices replayed and its saved markers on top (`load_tdt_recording`), splicing that re-fits the dF/F on what is left, events grouped by name, and the whole group analysis (`run_group_analysis`) repeatable from the settings file it saves. A script gets the numbers the app shows
- **Runs every analysis by name, and reproduces any result** — `run_analysis("event_peth", cache, {...})` returns tables whose columns match PyAT's CSV exports. Every export PyAT writes comes with a `.provenance.json` beside it (software versions, a fingerprint of the recording, its splices and signal names, the pipeline's own settings, the parameters; never recording data), and `python -m PyATCLI reproduce <record>` runs the analysis again without the app and checks the numbers against the export. `run`, `batch` and `group` do the same from a shell, so any figure from the GUI can be regenerated by a script
- **Works in float64, always** — TDT hands its streams over as float32, and numpy is not safe on single-precision data of a realistic length (`np.polyfit` silently loses the slope past ~3 minutes at 1 kHz). Every file load returns float64 and every public function that takes a signal widens its input, so nothing depends on a caller remembering to
- **Reports progress on anything slow** — loaders, the TDT pipeline, PETH and peak searches and the text-study pipelines take `progress=True` (a [tqdm](https://tqdm.github.io) bar on the console) or a callback receiving `(fraction, message)`, which is how PyAT shows a live percentage; silent by default

---

## Structure

```
PyATCLI/
  __init__.py              Public API — see below
  numeric.py                The float64 rule: every load, and every public function that takes a signal, works in float64
  models.py                  Parametric model functions and the curve-fit registry (CURVE_FIT_MODELS)
  progress.py                 tqdm-based progress reporting shared by the slow functions (Plan, track)
  _version.py                  Which library version is running (a source checkout reports its own)
  sources/                     Reading a recording from disk
    load.py                      load_dataset(), load_dataset_file(): detect the format, read it, return a Dataset (load_tdt, load_oxysoft ...)
    dataset.py                    Dataset struct, DataFormat enum, format detection
    processing_TDT.py              TDT tank reading, motion + bleach correction, dF/F, denoising, event markers
    processing_oxysoft.py           Oxysoft .txt reading: the Legend-block parser, folder and single-file readers
    tables.py                        Generic Excel/CSV/TSV/text parser with sub-table detection (load_any_file)
    pt2.py                            .pt2 EFNMR/MRI image parser
  recording.py                 A recording as PyAT holds it: saved state, building / splicing / loading it headlessly
  splice.py                     Non-destructive trim / cut-and-stitch of signals and markers
  markers.py                     How event markers are named and grouped by name
  saved_state.py                  Writing the saved state (markers, splices, signal names, highlights, text) — what Save Changes writes
  outputs.py                       Where results go under an Output folder: <type>/<recording> (Analysis)/, never overwriting
  analysis/                         One file per tool, each a pure function of arrays
    shared.py  zscore_peth.py  event_peth.py  auc.py  fft.py  peak_finder.py  curve_fit.py  intervals.py
    group/                            group analysis: the spec (__init__.py), the trials, the mixed models, the report,
                                      the scan of what each recording holds, and the headless run with its settings file
      trials.py  stats.py  report.py  scan.py  run.py
  analyses.py                   Every analysis by name: run_analysis(kind, cache, parameters) -> tables that mirror PyAT's CSV exports
  provenance.py                  How a result was produced, as data: software, the recording's fingerprint and state, pipeline settings, parameters
  reproduce.py                    Run an analysis again from its provenance record and compare it with an export
  results.py, viewplan.py          Reading a finished analysis back from its files (open_results), and what the main plot shows of a recording, as data (plan_view)
  cli/, __main__.py                 The command line: python -m PyATCLI (and the pyatcli command); docs.py writes its reference and man page
  ops/                               The operations the command line and the engine share: JSON in, JSON out
  engine/                            `serve`: the operations over a pipe, with recordings kept open (wire.py, store.py, session.py, server.py), and contract.py: what its callers can count on (ENGINE_CONTRACT.json)
  fieldstudy/                         The text-field study
    study.py                           Grouped-text-field study pipeline (embeddings, delta vector, paired similarity)
    validation.py                       Statistical validation for the paired-similarity metric (permutation test, Cohen's d, regression, bootstrap CI, leave-one-out)
```

The layers: `sources/` reads a format and computes (`processing_*`), and `sources/load.py` is the only place that ties format detection to the right reader and wraps its result into a `Dataset`; `recording.py` and `analysis/` work on plain arrays and dicts — the same functions PyAT calls, which is why a script reproduces what the app shows. There is no GUI and no plotting code in the library: it computes and writes numbers, and PyAT draws them (its `figures` package), so the library needs no matplotlib.

---

## Installation

```bash
pip install PyATCLI
```

That also installs the `pyatcli` command (`python -m PyATCLI` is the same thing). For the development version:

```bash
pip install git+https://github.com/zakgm2/PyATCLI.git
```

As a dependency of another project: `PyATCLI>=2026.10.10` in its `requirements.txt` or `pyproject.toml`.

### Requirements

- Python 3.10+
- `numpy`, `scipy`, `tdt`, `pandas`, `sentence-transformers`, `statsmodels`, `scikit-learn`, `tqdm` (installed automatically)
- `sentence-transformers` pulls in `torch`/`transformers` as transitive dependencies — a genuinely heavy install (hundreds of MB) if you only need the signal-processing side; only actually loaded when you call `embed_text_fields`/`run_field_study_pipeline`
- `openpyxl` — only needed for `.xlsx`/`.xls` files; imported lazily with a clear error if missing when you actually try to load Excel

---

## Usage

```python
import PyATCLI as pl

# Detect + load a TDT tank or Oxysoft export folder
fmt     = pl.detect_format(folder_path)
dataset = pl.load_dataset(folder_path, fmt)

# Or load a single Oxysoft .txt file directly
dataset = pl.load_dataset_file(file_path)

# Every loader returns the same universal Dataset struct
dataset.source_format   # "TDT" | "Oxysoft"
dataset.sample_rate      # Hz
dataset.signals            # (num_channels, num_samples)
dataset.channel_names        # list[str]
dataset.events                # [{'label': str, 'sample': int}, ...]
```

```python
# Anything slow takes progress=: True draws a tqdm bar, a callable gets (fraction 0-1, message)
result = pl.process_tdt_folder(folder_path, progress=True)
result = pl.process_tdt_folder(folder_path, progress=lambda fraction, message: print(f"{fraction:4.0%}  {message}"))
```

```python
# Generic tabular data (Excel/CSV/TSV/text) — returns one GenericTable per
# detected sub-table, since a single sheet can contain several side-by-side
tables = pl.load_any_file(path)
table  = tables[0]
table.headers   # list[str]
table.data      # (n_rows, n_cols) float64, NaN for missing

# Terranova .pt2 EFNMR/MRI image — returns a raw 2D array, not a Dataset
img = pl.load_pt2(path)   # (n, n) float64 (stored as float32 in the file)
```

```python
# A recording as PyAT holds it: processed to dF/F, its saved splices replayed, its saved markers on top
loaded = pl.load_tdt_recording(folder_path, regression_method="ols")
cache  = loaded["cache"]                        # the same dict the app keeps in ctx.cache
events = pl.group_markers_by_name(cache)         # {"L1P¹": [12.3, 45.6, ...], "Clap": [20.0], ...}

# Splice it — a TDT splice re-fits the dF/F on what is left, exactly as the app does
cut, applied = pl.apply_splices(cache, [{"mode": "cut_out", "start": 100.0, "end": 130.0}])

# A saved group analysis, run again from its settings file
spec, _ = pl.read_group_settings(group_folder)
results = pl.run_group_analysis(spec, out_dir=new_folder)
```

```python
# Any analysis PyAT offers, by name — the tables have the columns of PyAT's CSV exports
cache  = pl.load_recording(folder_path)["cache"]                 # TDT folder, Oxysoft .txt, or a table file
result = pl.run_analysis("event_peth", cache, {"event": "L1P¹", "pre_s": 10, "post_s": 20})
result.tables["mean_sem"]          # time_from_event_s, mean, sem
result.parameters                   # as run: defaults filled in, the event's times written out
pl.describe_kinds()                  # every analysis and every parameter

# Run one again from the record PyAT wrote beside an export, and check the numbers
rep = pl.reproduce_record("Event PETH_L1P_103522.provenance.json", compare=["Event PETH_L1P_103522.csv"])
rep.ok, rep.warnings
```

```bash
python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --param pre_s=10 --out out
python -m PyATCLI reproduce "out/Rat7_event_peth.provenance.json" --compare "out/Event PETH.csv"
python -m PyATCLI batch jobs.json --out batch-out        # many analyses over many recordings
```

`run_cli.py` in the repository folder starts the same command line without remembering the module name. Run it with nothing (or double-click it) and it opens Git Bash in that folder, shows the commands and leaves the prompt open, as right-click > "Git Bash Here" would; run it with arguments (`python run_cli.py info`) and it is the command line itself in the terminal you are in (`--terminal` and `--inline` choose; see the file's own docstring).

```python
opened = pl.open_results("out/Rec_event_peth.provenance.json")   # a finished analysis read back from its files: a record, a folder, a group's folder, a .csv
opened.kind, opened.parameters, opened.tables["trials"]       # what made it, what it was asked, and its tables (the library draws nothing; PyAT does)
```

```bash
python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --out out
python -m PyATCLI run event_peth "D:/rats/Rat7-250101" --param event="L1P¹" --output-folder Output   # into Output/TDT/Rat7-250101 (Analysis)/, as PyAT's exports
```

What the app saves with Save Changes — markers, splices, signal names, highlights, text — a script can save too, as the same files:

```python
pl.saved_state.add_marker("D:/rats/Rat7-250101", 312.5, label="Lever")
pl.saved_state.add_splice("D:/rats/Rat7-250101", "cut_out", 40.0, 55.0)       # stacks on the saved ones; replayed on every open
```

```bash
python -m PyATCLI saved marker "D:/rats/Rat7-250101" --time 312.5 --label Lever
python -m PyATCLI saved show "D:/rats/Rat7-250101"
python -m PyATCLI group setup "D:/cohort" --name Sucrose                  # lists the markers the recordings share, as the wizard does
python -m PyATCLI group setup "D:/cohort" --name Sucrose --marker "L1P¹" --marker "L2P¹" --output-folder Output --run
```

Every command takes `--json` (one document on standard output, so `... --json | jq` always works), `--plain` (tab-separated rows), `-q`, `-v`, `--no-input` and `--events` (progress as JSON lines on standard error); standard output is only ever the result and errors go to standard error with an exit status. `python -m PyATCLI` with no arguments lists the commands, and each command's `--help` has examples. [CLI_REFERENCE.md](CLI_REFERENCE.md) and the man page [man/pyatcli.1](man/pyatcli.1) are generated from the program (`python -m PyATCLI docs --format markdown|man`) and kept in step by a test. `python -m PyATCLI serve` runs the same operations as a long-lived engine speaking JSON lines on standard input and output, with recordings kept open between requests: it is what PyAT's window talks to, so none of the numerical work runs in the window's process ([the engine](API_REFERENCE.md#the-engine)).

See [API_REFERENCE.md](API_REFERENCE.md#running-analyses-by-name) for the analyses, the provenance record's layout and every command, and [Drawing results](API_REFERENCE.md#drawing-results) for where the figures went. The command line itself computes and writes files and never draws or opens a window: drawing is PyAT's.

```python
# Analysis
x_seg, z = pl.get_zscore_slice(time_array, signal, center_t, window=30)
freqs, power, seg_x, seg_y = pl.compute_fft_slice(time_array, signal, center_t, fs)
peaks = pl.find_fft_peaks(freqs, power)                  # [{"freq_hz", "power", "bpm"}, ...] — drawing them is up to you

# Curve fitting
result = pl.fit_model_to_segment(x_seg, y_seg, pl.single_exponential_model, p0_fn)
result["popt"], result["r2"], result["y_fit"]
```

```python
# Text field study — one JSON file per subject, e.g. P-0001.json. Pick
# pairs of fields to compare directly; no grouping concept needed.
fields = pl.peek_fields(folder_path)              # see what fields exist before picking pairs
df = pl.run_field_study_pipeline(
    folder_path,
    text_fields=["q1", "q2", "q3", "q4"],
    delta_pair=("q1", "q2"),                        # optional: how much did q2 change from q1
    paired_fields=[("q1", "q3", "pair1")],           # optional: does q1 track q3
)
# df has one row per subject: wordcount_<field>, low_quality_<field>, delta_magnitude,
# sim_<pair>, pvalue_<pair>, effect_size_<pair>, wc_confound_r_<pair>, ...
```

```python
# Statistical validation of the paired-similarity metric — one row per pair
summary = pl.run_validation_pipeline(
    folder_path,
    text_fields=["q1", "q3"],
    paired_fields=[("q1", "q3", "pair1")],
)
# summary: p_value, p_value_fdr, cohens_d, wc_coef_a/b + wc_pvalue_a/b,
# regression_r_squared, ci_lower/ci_upper, n_flagged_loo, flagged_participant_ids
```

See [FIELD_STUDY_METHODOLOGY.md](FIELD_STUDY_METHODOLOGY.md) for why each statistic in the
validation step is a sound, standard technique — useful if anyone asks.

See [CHANGELOG.md](CHANGELOG.md) for the version history.

---

## Public API

Everything importable from `PyATCLI` directly:

| Category | Names |
|----------|-------|
| Format detection | `detect_format`, `detect_format_file`, `DataFormat`, `Dataset` |
| Loading | `load_dataset`, `load_dataset_file`, `load_any_file`, `load_pt2` |
| TDT processing | `process_tdt_folder`, `compute_dff`, `validate_tdt_folder`, `get_tdt_struct`, `scan_tdt_markers`, `get_plot_data`, `correct_bleaching`, `denoise_signal`, `get_event_markers`, `debounce_events` |
| Oxysoft processing | `process_oxysoft_folder`, `process_oxysoft_file` |
| Saved state, written | the module `PyATCLI.saved_state`: `write_saved_markers`, `write_saved_splices`, `write_saved_signal_labels`, `write_saved_annotations`, `clear_saved_state`, `saved_state_summary`, `add_marker`, `remove_markers`, `add_splice`, `remove_splice`, `rename_signals`, `add_highlight`, `add_text` |
| Output folders | the module `PyATCLI.outputs`: `recording_dir`, `group_dir`, `unique_path`, `unique_stem`, `safe_folder_name` |
| Recordings and saved state | `load_recording`, `load_tdt_recording`, `find_tdt_recordings`, `make_tdt_cache`, `make_oxysoft_cache`, `make_generic_cache`, `build_signal_options`, `normalized_signal`, `read_saved_splices`, `read_saved_markers`, `read_saved_annotations`, `saved_state_dir`, `marker_sidecar_path`, `splice_sidecar_path`, `annotation_sidecar_path`, `legacy_marker_sidecar_path`, `signal_labels_path`, `read_saved_signal_labels`, `apply_signal_labels`, `current_signal_labels` |
| Running analyses | `run_analysis`, `kinds`, `describe_kinds`, `normalize_parameters`, `AnalysisResult`, `ParameterError`, `AnalysisError` |
| Reading results and views | `open_results`, `OpenedResults`, `NoResults`, `run_view`, `VIEW_KIND`; the modules `PyATCLI.results` and `PyATCLI.viewplan` (the library draws nothing: figures are PyAT's) |
| Provenance and reproducing | `build_record`, `write_record`, `read_record`, `recording_state`, `file_fingerprint`, `pipeline_parameters`, `software_versions`, `library_version`, `check_environment`, `markers_summary`, `output_entry`, `sidecar_path_for`, `PROVENANCE_SCHEMA_VERSION`, `PROVENANCE_SUFFIX`, `reproduce_record`, `compare_table`, `Reproduction`, `ReproduceError` (the module `PyATCLI.reproduce` holds the rest) |
| Splicing | `apply_splice`, `apply_splices`, `spliced_store_name`, `splice_keep_inside`, `splice_cut_out`, `replay_splices`, `MODE_CUT_OUT`, `MODE_KEEP_INSIDE` |
| Markers | `marker_display_label`, `store_display_name`, `group_markers_by_name`, `compute_marker_intervals` |
| Analysis | `get_zscore_slice`, `smooth_signal`, `bin_for_heatmap`, `compute_event_zscore_peth`, `compute_fft_slice`, `find_fft_peaks`, `compute_slope_segment`, `fit_model_to_segment`, `find_significant_peaks`, `find_peak_near_events`, `compute_auc_window`, `compute_auc_from_trace`, `compute_auc_matrix`, `compute_peri_event_from_trace`, `compute_peri_event_matrix`, `compute_group_stats`, `estimate_sample_rate`, `mean_channels` |
| Group analysis | `GroupSpec`, `scan_recording`, `scan_recordings`, `default_subject_names`, `load_group_recording`, `run_group_analysis`, `write_group_settings`, `read_group_settings`, `GROUP_SETTINGS_FILENAME`, `marker_index`, `common_markers`, `design_summary`, `extract_group_trials`, `fit_group_models`, `GroupResults`, `write_group_results`, `group_report_text` |
| Curve fit models | `CURVE_FIT_MODELS`, `linear_model`, `single_exponential_model`, `exponential_rise_model`, `double_exponential_model`, `gaussian_model`, `sinusoidal_model`, `visibility_model` |
| Text field study | `run_field_study_pipeline`, `load_field_study_folder`, `peek_fields`, `flag_low_quality`, `embed_text_fields`, `compute_delta_vector`, `compute_paired_similarity`, `permutation_test_similarity`, `wordcount_confound_check` |
| Field study validation | `run_validation_pipeline`, `build_validation_summary`, `cohens_d`, `benjamini_hochberg`, `wordcount_controlled_regression`, `bootstrap_mean_ci`, `leave_one_out_sensitivity` |

---

## License

AGPL-3.0 — see [LICENSE](LICENSE) for details.
