Metadata-Version: 2.4
Name: mat73-reader
Version: 0.1.0
Summary: Read MATLAB v7.3 HDF5 .mat files that scipy.io.loadmat cannot handle.
Author-email: William Garrow <williamgarrow@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: hdf5,mat73,matlab,scientific-data,scipy
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.9
Requires-Dist: h5py>=3.0
Requires-Dist: numpy>=1.20
Requires-Dist: pandas>=1.3
Provides-Extra: dev
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# mat73-reader

Read MATLAB v7.3 HDF5 `.mat` files in Python. **including MATLAB table objects** that other tools cannot decode.

## The MATLAB Table Problem

MATLAB v7.3 stores table objects using an undocumented internal system called MCOS (MATLAB Class Object System). Every existing Python tool (`scipy.io.loadmat()`, `mat73`, `hdf5storage`) fails on them:

```python
# scipy can't even open v7.3 files
>>> scipy.io.loadmat("experiment.mat")
NotImplementedError: Please use HDF reader for matlab v7.3 files

# mat73 opens the file but returns None for every table
>>> import mat73
>>> data = mat73.loadmat("experiment.mat")
ERROR: MATLAB type not supported: table, (uint32)   # x 800
>>> data["task"]["gaze"][0]
None
```

Tables are used extensively in neuroscience, signal processing, cognitive science, biomechanics, and clinical research datasets. If your `.mat` file contains tables, **mat73-reader is currently the only Python tool that can read them.**

```python
>>> from mat73_reader import load
>>> data = load("experiment.mat")
>>> data["task"]["gaze"][0]
      gaze_timestamp  world_index  confidence  norm_pos_x  norm_pos_y  ...
0        5410.551714          0.0    0.999499    0.446264    0.846886  ...
1        5410.555834          0.0    0.999653    0.446534    0.847007  ...
2        5410.559773          0.0    0.999648    0.446660    0.846410  ...
...
[8205 rows x 21 columns]
```

## How It Works

When other tools encounter a MATLAB table, they see a `(1,6) uint32` header and stop. mat73-reader decodes the MCOS block structure to follow the reference chain to the actual data:

```mermaid
graph TD
    subgraph "What other tools see"
        A["Table Header<br/>(1,6) uint32<br/>0xDD000000 ..."] -->|"???"| B["None"]
    end

    subgraph "What mat73-reader decodes"
        H["Table Header<br/>(1,6) uint32"] -->|"instance index"| M["MCOS Reference Array<br/>#subsystem#/MCOS"]
        M -->|"block offset + 0"| D["Column Data Refs<br/>(ncols, 1) object"]
        M -->|"block offset + 5"| N["Column Name Refs<br/>(ncols, 1) object"]
        D -->|"dereference"| D1["timestamp<br/>float64 (1, N)"]
        D -->|"dereference"| D2["confidence<br/>float64 (1, N)"]
        D -->|"dereference"| D3["...<br/>float64 (1, N)"]
        N -->|"dereference"| N1["'gaze_timestamp'<br/>uint16 chars"]
        N -->|"dereference"| N2["'confidence'<br/>uint16 chars"]
        N -->|"dereference"| N3["'...'<br/>uint16 chars"]
        D1 & D2 & D3 & N1 & N2 & N3 -->|"assemble"| DF["pandas DataFrame"]
    end

    style B fill:#ff6b6b,color:#fff
    style DF fill:#51cf66,color:#fff
```

Each table instance occupies a fixed block of 7 consecutive entries in the MCOS reference array:

```
Block layout (7 slots per table):
  +0  (ncols, 1) object refs   --> column data arrays (float64, int, etc.)
  +1  (1, 1) float64           --> ndims
  +2  (1, 1) float64           --> nrows
  +3  (2,) uint64              --> segment info
  +4  (1, 1) float64           --> nvars (number of columns)
  +5  (ncols, 1) object refs   --> column name strings (uint16-encoded)
  +6  Group                    --> table properties (units, descriptions, etc.)
```

The instance index from the table header maps to a block offset:

```
block_start = 2 + (instance - 1) * 7
```

This structure is not documented by MathWorks. It was reverse-engineered by analyzing real-world scientific datasets.

## Real-World Validation

mat73-reader has been validated against the [COLET dataset](https://zenodo.org/records/7766785) (Cognitive workLoad Estimation via Eye-Tracking), a 3.8 GB MATLAB v7.3 file containing:

- 47 subjects, 4 tasks per subject
- 4 data fields per task (gaze, pupil, blinks, annotation)
- **752 MATLAB table objects** total
- Over 14,000 individual data arrays

Every table was successfully decoded into a pandas DataFrame with correct column names and data types. Other Python tools return `None` for all 752 tables.

## Installation

```bash
pip install mat73-reader
```

Or install from source:

```bash
git clone https://github.com/WilliamGarrow/mat73-reader.git
cd mat73-reader
pip install -e ".[dev]"
```

## Usage

### Python API

```python
from mat73_reader import load, inspect

# Inspect file contents without loading data
variables = inspect("experiment.mat")
for var in variables:
    print(var)

# Load everything
data = load("experiment.mat")

# Load a specific top-level variable
results = load("experiment.mat", variable="results")

# Force all compatible arrays to pandas DataFrames
data = load("experiment.mat", as_dataframe=True)
```

MATLAB tables are always returned as DataFrames automatically, no flags needed.

### Command Line

```bash
# List variables, types, and shapes
mat73-reader inspect experiment.mat

# Convert to CSV (one file per variable)
mat73-reader convert experiment.mat --format csv --output ./csv_output/

# Convert to JSON
mat73-reader convert experiment.mat --format json --output experiment.json

# Convert a single variable
mat73-reader convert experiment.mat --variable gaze_data --format csv
```

## What It Handles

| MATLAB Type | Python Type | Notes |
|------------|-------------|-------|
| **Table objects** | **`pandas.DataFrame`** | **Column names and data types preserved** |
| Numeric arrays | `numpy.ndarray` | Transposed to row-major order |
| Structs | `dict` | Nested to arbitrary depth |
| Cell arrays | `list` | HDF5 object references resolved |
| Char arrays | `str` | Decoded from uint16 |
| Scalars | Python `int`/`float` | Single-element arrays squeezed |

## When to Use This vs. Other Tools

| Scenario | Tool |
|----------|------|
| `.mat` v5 or earlier (no tables) | `scipy.io.loadmat()` |
| `.mat` v7.3 with arrays and structs only | `mat73` or **mat73-reader** |
| `.mat` v7.3 with **table objects** | **mat73-reader** (only option in Python) |
| Not sure what format you have | Try `mat73-reader` first; it will tell you if it's not v7.3 |

## Development

```bash
git clone https://github.com/WilliamGarrow/mat73-reader.git
cd mat73-reader
python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
```

### Test Suite

38 tests covering:
- Standard v7.3 reading (arrays, structs, cell arrays, char arrays, scalars)
- MCOS table header detection (positive and negative cases)
- Single and multi-table decoding with synthetic fixtures
- Column name extraction and data value verification
- Edge cases (non-table uint32 arrays, missing variables, invalid files)

## License

Apache 2.0. See [LICENSE](LICENSE) for details.
