Metadata-Version: 2.5
Name: mat73-reader
Version: 0.1.1
Summary: Read MATLAB v7.3 HDF5 .mat files that scipy.io.loadmat cannot handle.
Author-email: William Garrow <williamgarrow@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: hdf5,mat73,matlab,scientific-data,scipy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Requires-Dist: h5py>=3.0
Requires-Dist: numpy>=1.20
Requires-Dist: pandas>=1.3
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# mat73-reader

Read MATLAB v7.3 HDF5 `.mat` files in Python. **including MATLAB table objects** that other tools cannot decode.

## The MATLAB Table Problem

MATLAB v7.3 stores table objects using an undocumented internal system called MCOS (MATLAB Class Object System). Every existing Python tool (`scipy.io.loadmat()`, `mat73`, `hdf5storage`) fails on them:

```python
# scipy can't even open v7.3 files
>>> scipy.io.loadmat("experiment.mat")
NotImplementedError: Please use HDF reader for matlab v7.3 files

# mat73 opens the file but returns None for every table
>>> import mat73
>>> data = mat73.loadmat("experiment.mat")
ERROR: MATLAB type not supported: table, (uint32)   # x 800
>>> data["task"]["gaze"][0]
None
```

Tables are used extensively in neuroscience, signal processing, cognitive science, biomechanics, and clinical research datasets. If your `.mat` file contains tables, **mat73-reader reads what `scipy` and `mat73` cannot**, with a tables-first API and a CLI. (See [FORMAT.md](FORMAT.md) for the reverse-engineered format notes and the other projects that have independently mapped MCOS.)

```python
>>> from mat73_reader import load
>>> data = load("experiment.mat")
>>> data["task"]["gaze"][0]
      gaze_timestamp  world_index  confidence  norm_pos_x  norm_pos_y  ...
0        5410.551714          0.0    0.999499    0.446264    0.846886  ...
1        5410.555834          0.0    0.999653    0.446534    0.847007  ...
2        5410.559773          0.0    0.999648    0.446660    0.846410  ...
...
[8205 rows x 21 columns]
```

## How It Works

When other tools encounter a MATLAB table, they see a `(1,6) uint32` header and stop. mat73-reader decodes the MCOS block structure to follow the reference chain to the actual data:

```mermaid
graph TD
    subgraph "What other tools see"
        A["Table Header<br/>(1,6) uint32<br/>0xDD000000 ..."] -->|"???"| B["None"]
    end

    subgraph "What mat73-reader decodes"
        H["Table Header<br/>(1,6) uint32"] -->|"instance index"| M["MCOS Reference Array<br/>#subsystem#/MCOS"]
        M -->|"block offset + 0"| D["Column Data Refs<br/>(ncols, 1) object"]
        M -->|"block offset + 5"| N["Column Name Refs<br/>(ncols, 1) object"]
        D -->|"dereference"| D1["timestamp<br/>float64 (1, N)"]
        D -->|"dereference"| D2["confidence<br/>float64 (1, N)"]
        D -->|"dereference"| D3["...<br/>float64 (1, N)"]
        N -->|"dereference"| N1["'gaze_timestamp'<br/>uint16 chars"]
        N -->|"dereference"| N2["'confidence'<br/>uint16 chars"]
        N -->|"dereference"| N3["'...'<br/>uint16 chars"]
        D1 & D2 & D3 & N1 & N2 & N3 -->|"assemble"| DF["pandas DataFrame"]
    end

    style B fill:#ff6b6b,color:#fff
    style DF fill:#51cf66,color:#fff
```

Each table instance occupies a fixed block of 7 consecutive entries in the MCOS reference array:

```
Block layout (7 slots per table):
  +0  (ncols, 1) object refs   --> column data arrays (float64, int, etc.)
  +1  (1, 1) float64           --> ndims
  +2  (1, 1) float64           --> nrows
  +3  (2,) uint64              --> segment info
  +4  (1, 1) float64           --> nvars (number of columns)
  +5  (ncols, 1) object refs   --> column name strings (uint16-encoded)
  +6  Group                    --> table properties (units, descriptions, etc.)
```

The instance index from the table header maps to a block offset:

```
block_start = 2 + (instance - 1) * 7
```

This structure is not documented by MathWorks. It was reverse-engineered by analyzing real-world scientific datasets.

## Real-World Validation

mat73-reader has been validated against the [COLET dataset](https://zenodo.org/records/7766785) (Cognitive workLoad Estimation via Eye-Tracking), a 3.8 GB MATLAB v7.3 file containing:

- 47 subjects, 4 tasks per subject
- 4 data fields per task (gaze, pupil, blinks, annotation), plus one subject-info table per subject
- **799 MATLAB table objects** total, 40 of them empty (blink tables for tasks with no blinks)
- Over 14,000 individual data arrays

Every table decodes into a pandas DataFrame with correct column names and data types. In August 2026 the output was cross-checked column for column (200 million cells) against an independent MCOS decoder; the only disagreement was the 40 empty tables, which version 0.1.0 read as two rows of `[0, 1]`. Fixed in 0.1.1. Other Python tools return `None` for all 799 tables.

## Installation

```bash
pip install mat73-reader
```

Or install from source:

```bash
git clone https://github.com/WilliamGarrow/mat73-reader.git
cd mat73-reader
pip install -e ".[dev]"
```

## Usage

### Python API

```python
from mat73_reader import load, inspect

# Inspect file contents without loading data
variables = inspect("experiment.mat")
for var in variables:
    print(var)

# Load everything
data = load("experiment.mat")

# Load a specific top-level variable
results = load("experiment.mat", variable="results")

# Force all compatible arrays to pandas DataFrames
data = load("experiment.mat", as_dataframe=True)
```

MATLAB tables are always returned as DataFrames automatically, no flags needed.

### Command Line

```bash
# List variables, types, and shapes
mat73-reader inspect experiment.mat

# Convert to CSV (one file per variable)
mat73-reader convert experiment.mat --format csv --output ./csv_output/

# Convert to JSON
mat73-reader convert experiment.mat --format json --output experiment.json

# Convert a single variable
mat73-reader convert experiment.mat --variable gaze_data --format csv
```

## What It Handles

| MATLAB Type | Python Type | Notes |
|------------|-------------|-------|
| **Table objects** | **`pandas.DataFrame`** | **Column names and data types preserved** |
| Numeric arrays | `numpy.ndarray` | Transposed to row-major order |
| Structs | `dict` | Nested to arbitrary depth |
| Cell arrays | `list` | HDF5 object references resolved |
| Char arrays | `str` | Decoded from uint16 |
| Scalars | Python `int`/`float` | Single-element arrays squeezed |

## When to Use This vs. Other Tools

| Scenario | Tool |
|----------|------|
| `.mat` v5 or earlier (no tables) | `scipy.io.loadmat()` |
| `.mat` v7.3 with arrays and structs only | `mat73` or **mat73-reader** |
| `.mat` v7.3 with **table objects** | **mat73-reader** (tables-first, CLI) or [matio](https://github.com/foreverallama/matio) (broad MCOS object support) |
| Not sure what format you have | Try `mat73-reader` first; it will tell you if it's not v7.3 |

## FAQ (from the launch thread)

**Why not Octave?** Octave's loader stops at v7, so it cannot open v7.3 HDF5
files, and it has no `table` class to reconstruct into. For Octave users the
practical path is `mat73-reader convert file.mat --to csv`.

**Why not h5py?** h5py opens the container fine; it is the MCOS object layer
it cannot interpret. You get opaque uint32 references into a hidden group.
Following that mapping is exactly what this library does; the mapping itself
is written up in [FORMAT.md](FORMAT.md).

**Can't MATLAB just export a CSV?** Yes: `writetable` is the right answer
when you have MATLAB. This tool is for the other case: a shared dataset, no
license, tables already locked in v7.3.

## Development

```bash
git clone https://github.com/WilliamGarrow/mat73-reader.git
cd mat73-reader
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

### Test Suite

38 tests covering:
- Standard v7.3 reading (arrays, structs, cell arrays, char arrays, scalars)
- MCOS table header detection (positive and negative cases)
- Single and multi-table decoding with synthetic fixtures
- Column name extraction and data value verification
- Edge cases (non-table uint32 arrays, missing variables, invalid files)

## License

Apache 2.0. See [LICENSE](LICENSE) for details.
