Metadata-Version: 2.1
Name: etformat
Version: 1.3.0
Summary: A Python package for eye-tracking data processing
Home-page: https://github.com/Aaronkhodami/etformat
Author: Aaron Khodami
Author-email: aaron.a.khodami@gmail.com
Project-URL: Homepage, https://aaronkhodami.org
Project-URL: Source, https://github.com/Aaronkhodami/etformat
Project-URL: Documentation, https://aaronkhodami.org/etformat/
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: matplotlib

# etformat — Eye-Tracking Data Processing

**etformat** is a Python package for extracting, converting, cleaning, and analyzing eye-tracking data recorded with SR Research EyeLink trackers (`.edf` files). It reads EDF files directly (via the EyeLink Developers Kit) and turns them into tidy pandas DataFrames / CSV files, then gives you ready-made tools for per-trial statistics, saccade/fixation/blink summaries, calibration quality checks, and gaze/saccade plotting.

Typical uses: psychology, neuroscience, and usability research where you need to go from a raw `.edf` recording to analysis-ready tables with as little boilerplate as possible.

## Features

- **`export()`** — convert an EDF file into `*_events.csv` and `*_samples.csv`
- **`EDFFile`** — load an EDF file directly into DataFrames (`events`, `samples`, `recordings`, plus parsed `saccades`, `fixations`, `blinks`, `variables`, `triggers`, `aois`)
- **`edfinfo()`** — print recording metadata (sampling rate, screen size, calibration type, EyeLink version, recording date, …)
- **`extract_channels()`** — list the data channels actually present for the recorded eye(s)
- **`describe()`** — quick per-trial or all-trial fixation/saccade/blink counts
- **`report()`** — detailed, tested per-trial QC report (durations, sample rate, valid-gaze percentage, saccade amplitudes, …)
- **`clean()`** — drop unassigned rows and empty/all-zero columns from samples/events
- **`saccade_amplitude_average()`** — average saccade amplitude (in degrees) per trial
- **`calibration()`** — parse and plot calibration/validation results
- **`plot_gaze()`** / **`plot_saccades()`** — quick-look plots of a trial's gaze path or saccades

## Installation

### 1. Install the EyeLink Developers Kit (required)

etformat reads `.edf` files through SR Research's `edfapi` library, which ships with the **EyeLink Developers Kit**. This is a prerequisite — etformat will not work without it.

(© SR Research & Alexander (Sasha) Pastukhov)

1. Register at [www.sr-research.com/support](https://www.sr-research.com/support) and wait for account activation.
2. Download the **EyeLink Developers Kit / API** for your platform from *SR Support Forum › Downloads › EyeLink Developers Kit / API*.
3. Install it using the default location for your OS:

   | OS | Default `edfapi` location |
   |----|----------------------------|
   | Windows | `C:/Program Files (x86)/SR Research/EyeLink/libs/x64` |
   | macOS | `/Library/Frameworks` |
   | Linux | `/usr/lib` (already on the global path) |

If you installed to a non-default location, either add it to `PATH`, set the `EDFAPI_LIB` environment variable, or pass `libpath=...` to `EDFFile`/`export`.

### 2. Install etformat

```bash
pip install etformat
```

Or, for development (editable install from a clone of this repo):

```bash
git clone https://github.com/Aaronkhodami/etformat.git
cd etformat
pip install -e .
```

Dependencies (`numpy`, `pandas`, `matplotlib`) are installed automatically.

## Quick start

```python
import etformat as et

# 1. Convert an EDF file to CSV (events + samples, saved next to the EDF file)
paths = et.export("recording.edf")
print(paths)
# {'events': '.../recording_events.csv', 'samples': '.../recording_samples.csv'}

# 2. Print recording metadata
et.edfinfo("recording.edf")

# 3. Build a per-trial QC report
report = et.report(paths["events"])  # samples file is found automatically

# 4. Plot gaze and saccades for trial 3
import pandas as pd
samples = pd.read_csv(paths["samples"])
events = pd.read_csv(paths["events"])
et.plot_gaze(samples, trial_number=3, screen_width=1920, screen_height=1080)
et.plot_saccades(events, trial_number=3, screen_width=1920, screen_height=1080)
```

A sample recording is included at [`tests/75936282.edf`](tests/75936282.edf) if you want to try the package end to end without your own file.

## Function reference

### `export(edf_filepath, **kwargs)`

Converts an EDF file into two CSV files saved next to it: `<name>_events.csv` and `<name>_samples.csv`. Returns a dict with the paths that were written.

```python
import etformat as et

paths = et.export("test.edf")
# Events saved to: .../test_events.csv
# Samples saved to: .../test_samples.csv
```

Useful options (passed as keyword arguments, see `EDFFile` below for the full list):

```python
et.export(
    "test.edf",
    start_marker_string="TRIALID",   # message prefix marking trial start
    end_marker_string=None,          # None -> a trial stays "active" until the next start marker
    parse_events="all",              # or e.g. ["saccades", "fixations"]
    verbose=True,                    # show a progress bar while parsing
)
```

Missing measurements are written as empty CSV cells (`NaN` in pandas) rather than dropped rows, so blink timing and the continuous sample timeline are preserved. The EDF API's floating-point "missing data" sentinel (`100000000.0`) is masked out, and fields are masked per-sample according to which eye/data type was actually recorded — so left-only, right-only, and binocular recordings all export cleanly through the same path.

### `EDFFile` — direct, lower-level access

For anything beyond a straight CSV export, load the file directly:

```python
from etformat import EDFFile

edf = EDFFile("test.edf", loadevents=True, loadsamples=True, parse_events="all")

edf.events       # all parsed events (includes sttime_rel / entime_rel relative to recording start)
edf.samples      # all samples (includes time_rel relative to recording start)
edf.recordings   # recording start/end blocks and metadata (sample rate, eye, pupil type, ...)
edf.preamble     # raw EDF preamble text

# Convenience tables parsed from `events` (present when requested via parse_events="all" or a list):
edf.saccades     # ENDSACC events with a computed `duration` column
edf.fixations    # ENDFIX events with a computed `duration` column
edf.blinks       # ENDBLINK events with a computed `duration` column
edf.variables    # TRIAL_VAR messages, pivoted to one row per trial
edf.triggers     # messages starting with the trigger_marker (default "TRIGGER")
edf.aois         # "!V IAREA RECTANGLE" area-of-interest definitions
```

Key constructor options:

| Parameter | Default | Description |
|---|---|---|
| `consistency` | `2` | Timestamp consistency check: `0` none, `1` check & report, `2` check & fix |
| `loadevents` / `loadsamples` | `True` / `False` | Whether to load each table |
| `sample_fields` | `None` | Restrict which sample fields are populated |
| `start_marker_string` | `"TRIALID"` | Message prefix marking a trial's start; `None` disables trial numbering |
| `end_marker_string` | `"TRIAL_RESULT"` | Message prefix marking a trial's end; `None` keeps a trial active until the next start marker |
| `parse_events` | `"all"` | Which of `saccades`, `fixations`, `blinks`, `variables`, `triggers`, `aois` to parse |
| `wide_variables` | `True` | Pivot `TRIAL_VAR` messages to one row per trial instead of long format |
| `trigger_marker` | `"TRIGGER"` | Message prefix identifying trigger events |
| `verbose` | `False` | Show a progress bar while parsing |
| `libpath` | `None` | Explicit path to the `edfapi` library |

### `edfinfo(edf_file_path)`

Prints a human-readable summary of the recording, parsed straight from the EDF preamble and metadata:

```python
et.edfinfo("test.edf")
```
```
** EDF File Information **
==================================================
Eye Recorded      : LEFT
Sampling Rate     : 1000.0 Hz
Pupil Measurement : AREA
Recording Mode    : CR
Data Type         : SAMPLES and EVENTS
Calibration Type  : HV9
Screen Size       : 2560 x 1440 pixels
EyeLink Version   : EYELINK CL
Camera Version    : EyeLink USBCAM Version 1.01
Serial Number     : CLU-DAF19
Recorded By       : ...
Recording Date    : ...
==================================================
```

### `extract_channels(edf_file_path)`

Prints the sample channels actually populated for the recorded eye(s) (e.g. only `*L` columns for a left-eye-only recording).

### `describe(data_source, trial_number=None)`

Quick counts and averages per trial. Accepts either an EDF file path or a `(events_df, samples_df)` tuple. Returns a dict for a single trial or a DataFrame for all trials.

```python
et.describe("test.edf")                 # DataFrame, one row per trial
et.describe("test.edf", trial_number=3) # dict for trial 3
et.describe((events_df, samples_df))    # reuse already-loaded DataFrames
```

Each row/dict includes: `Total Duration (ms)`, `Total Samples`, `Number of Fixations`, `Number of Saccades`, `Number of Blinks`, `Avg Fixation Duration (ms)`, `Avg Saccade Amplitude`.

### `report(events, output_path=None, samples=None, ...)`

A more rigorous, unit-tested per-trial QC report than `describe()`. Works from CSV paths or DataFrames, and computes trial boundaries from message markers (falling back to the first/last assigned sample when markers are absent).

```python
# From exported CSVs — the matching *_samples.csv is found automatically
# because it sits next to a file named "<name>_events.csv"
report_path = et.report("test_events.csv")   # -> saves "test_report.csv", returns its path

# From DataFrames — returns the report DataFrame instead of saving a file
report_df = et.report(events_df, samples=samples_df)
```

Each row reports: `trial_start`, `trial_end`, `recording_duration`, `sample_rate_hz`, `recorded_eye`, `total_samples`, `total_events`, `num_fixations`/`num_saccades`/`num_blinks` and their total/average durations, `avg_saccade_amplitude` (in degrees, using each saccade's own units-per-degree), `num_messages`, and gaze-quality columns: `valid_gaze_left_samples`, `valid_gaze_right_samples`, `valid_gaze_samples`, `missing_gaze_samples`, `valid_gaze_percent`, `zero_pupil_samples`.

Trial-boundary detection is configurable:

```python
et.report(
    "test_events.csv",
    start_marker_string="TRIALID",     # default
    end_marker_string="TRIAL_RESULT",  # default; also recognizes "Trial N ended" messages
    # end_marker_string=None disables message-based end detection entirely,
    # using each trial's last assigned sample instead
)
```

### `clean(samples, events, verbose=True, copy=False)`

Drops rows with no trial assigned and columns that are entirely empty or all-zero (e.g. the unrecorded eye's columns, or a head-tracker that wasn't used). Accepts CSV paths or DataFrames, matching the input type on return.

```python
# DataFrames in, cleaned DataFrames out
clean_samples, clean_events = et.clean(samples_df, events_df)

# File paths in: overwrite in place...
et.clean("test_samples.csv", "test_events.csv")

# ...or write "*_cleaned.csv" copies instead
s_path, e_path = et.clean("test_samples.csv", "test_events.csv", copy=True)
```

### `saccade_amplitude_average(events, output_path=None)`

Average saccade amplitude (degrees) per trial, computed from each saccade's start/end gaze position and its own units-per-degree conversion.

```python
et.saccade_amplitude_average("test_events.csv")   # saves "test_events_average_saccade_amplitude.csv"
et.saccade_amplitude_average(events_df)           # returns a DataFrame instead
```

### `calibration(edf_file_path)`

Parses `!CAL` calibration/validation messages from the EDF file, prints the calibration model and average/max validation error per eye, and shows a scatter plot of validation points annotated with their offsets (in degrees).

```python
et.calibration("test.edf")
```

### `plot_gaze(samples, trial_number, screen_width=2560, screen_height=1440)`

Plots the continuous gaze path for one trial. Automatically picks the eye that actually has data (left, right, or defaults to right when both eyes were recorded); pass your real screen resolution to avoid axis scaling that doesn't match your setup.

```python
et.plot_gaze(samples_df, trial_number=3, screen_width=1920, screen_height=1080)
```

### `plot_saccades(events, trial_number=0, screen_width=2560, screen_height=1440)`

Plots numbered, arrowed saccade vectors (start → end gaze position) for one trial.

```python
et.plot_saccades(events_df, trial_number=3, screen_width=1920, screen_height=1080)
```

## Output CSV columns

### Sample File (`*_samples.csv`)

| Column | Description | Note |
|--------|-------------|------|
| **trial** | Trial index | Added, not from EDF file |
| **time** | Timestamp of the sample (in milliseconds) | |
| **time_rel** | Time relative to recording start | Added |
| **pxL/pxR** | Left/Right eye pupil X position | |
| **pyL/pyR** | Left/Right eye pupil Y position | |
| **hxL/hxR** | Left/Right eye head-ref X coordinate | |
| **hyL/hyR** | Left/Right eye head-ref Y coordinate | |
| **paL/paR** | Left/Right eye pupil size or area | |
| **gxL/gxR** | Left/Right eye gaze X coordinate (screen) | |
| **gyL/gyR** | Left/Right eye gaze Y coordinate (screen) | |
| **rx** | Pixels per degree on X axis | |
| **ry** | Pixels per degree on Y axis | |
| **gxvelL/gxvelR** | Left/Right eye gaze X velocity | |
| **gyvelL/gyvelR** | Left/Right eye gaze Y velocity | |
| **hxvelL/hxvelR** | Left/Right eye head-ref X velocity | |
| **hyvelL/hyvelR** | Left/Right eye head-ref Y velocity | |
| **rxvelL/rxvelR** | Left/Right eye raw X velocity | |
| **ryvelL/ryvelR** | Left/Right eye raw Y velocity | |
| **fgxvelL/fgxvelR** | Left/Right eye fast gaze X velocity | |
| **fgyvelL/fgyvelR** | Left/Right eye fast gaze Y velocity | |
| **fhxvelL/fhxvelR** | Left/Right eye fast head-ref X velocity | |
| **fhyvelL/fhyvelR** | Left/Right eye fast head-ref Y velocity | |
| **frxvelL/frxvelR** | Left/Right eye fast raw X velocity | |
| **fryvelL/fryvelR** | Left/Right eye fast raw Y velocity | |
| **hdata0-hdata7** | Head-tracker data channels | 8 values |
| **flags** | Flags indicating what data is present | |
| **input** | Extra input value | |
| **buttons** | Button states and changes | |
| **htype** | Head-tracker type (0 = none) | |
| **errors** | Processing error flags | |

Missing measurements are written as empty CSV cells (`NaN` when read by
pandas). No sample rows are removed. This preserves blink and data-loss timing
while preventing the EDF API floating-point sentinel (`100000000.0`) from being
treated as a real gaze position or velocity. Eye-specific and optional fields
are masked according to each sample's `flags`, so left-only, right-only, and
binocular recordings use the same export path.

### Events File (`*_events.csv`)

| Column | Description | Note |
|--------|-------------|------|
| **trial** | Trial index | Added |
| **time** | Time of the event | |
| **type** | Type of event (e.g. blink, saccade) | |
| **read** | Flags for which data is present | |
| **sttime** | Start time of the event | |
| **sttime_rel** | Start time relative to recording start | Added |
| **entime** | End time of the event | |
| **entime_rel** | End time relative to recording start | Added |
| **duration** | Event duration | Computed |
| **hstx** | Head-ref X at start | |
| **hsty** | Head-ref Y at start | |
| **gstx** | Gaze X at start | |
| **gsty** | Gaze Y at start | |
| **sta** | Pupil size at start | |
| **henx** | Head-ref X at end | |
| **heny** | Head-ref Y at end | |
| **genx** | Gaze X at end | |
| **geny** | Gaze Y at end | |
| **ena** | Pupil size at end | |
| **havx** | Average head-ref X | |
| **havy** | Average head-ref Y | |
| **gavx** | Average gaze X | |
| **gavy** | Average gaze Y | |
| **ava** | Average pupil size | |
| **avel** | Average velocity | |
| **pvel** | Peak velocity | |
| **svel** | Start velocity | |
| **evel** | End velocity | |
| **supd_x** | Start units-per-degree (X axis) | |
| **eupd_x** | End units-per-degree (X axis) | |
| **supd_y** | Start units-per-degree (Y axis) | |
| **eupd_y** | End units-per-degree (Y axis) | |
| **eye** | Eye used | 0 = left, 1 = right |
| **status** | Event status flags | |
| **flags** | Additional flags | |
| **input** | Input state | |
| **buttons** | Button states | |
| **parsedby** | Who parsed this event | |
| **message** | Message string (if any) | |

### Recording Data (`edf.recordings`)

| Column | Description | Values |
|--------|-------------|--------|
| **time** | Time of recording start or end | |
| **sample_rate** | Sampling rate | Hz |
| **eflags** | Event-related flags | |
| **sflags** | Sample-related flags | |
| **state** | Recording state | 0 = END, 1 = START |
| **record_type** | What was recorded | 1 = samples, 2 = events, 3 = both |
| **pupil_type** | Pupil measurement | 0 = area, 1 = diameter |
| **recording_mode** | Recording mode | 0 = pupil only, 1 = corneal reflection |
| **filter_type** | Filter applied | 1, 2, or 3 |
| **pos_type** | Position type | 0 = gaze, 1 = HREF, 2 = raw |
| **eye** | Eye recorded | 1 = left, 2 = right, 3 = both |

## Running the tests

```bash
pip install pytest
pytest tests/
```

The test suite covers the EDF struct layout, sample/event parsing edge cases (missing data, binocular masking, relative timing), and the `report()` calculations — it does not require the EyeLink Developers Kit to run.

## Documentation

For more detail and worked examples (calibration, cleaning, describing, exporting, plotting, reporting), see the full documentation:
📖 **[etformat Documentation](https://aaronkhodami.org/etformat/)**

## License

MIT — see [LICENSE](LICENSE).
