Metadata-Version: 2.4
Name: etiket_sync_agent_quantify
Version: 0.3.0b1
Summary: Quantify backend for eTiKeT sync agent
Author: QHarbor team
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://qharbor.nl
Project-URL: Documentation, https://docs.qharbor.nl
Keywords: etiket,sync,backend,quantify,quantum
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: etiket_sync_agent>=0.3.0b1
Requires-Dist: xarray
Provides-Extra: test
Requires-Dist: quantify-core>=0.7; extra == "test"
Dynamic: license-file

# eTiKeT Sync Agent - Quantify Backend

Backend for synchronizing Quantify datasets with the eTiKeT platform. This backend scans Quantify data directories and syncs HDF5 datasets to the cloud.

## Installation

```bash
pip install etiket_sync_agent_quantify
```

The package is automatically discovered by `etiket_sync_agent` through the entry-point system.

## What Gets Synchronized

When a Quantify dataset is synced, the following data is extracted and uploaded:

| Quantify Data | eTiKeT Field | Description |
|---------------|--------------|-------------|
| TUID (first 26 chars) | `alt_uid` | Unique identifier extracted from the folder name |
| Folder name (after TUID) | `name` | Name of the measurement |
| TUID timestamp | `collected` | Measurement timestamp parsed from TUID format |
| Variable `long_name`/`name` attrs | `tags` | Extracted from HDF5 dataset attributes |
| Config `set_up` | `attributes.set-up` | Experimental setup from configuration |
| HDF5 files | Data files | xarray datasets converted and uploaded |
| JSON/text files | Auxiliary files | Additional files in the dataset folder |

### Data Processing

- **Grid conversion**: Datasets are automatically converted to gridded format for efficient visualization
- **Standard deviation detection**: Variables ending with `_u` are marked as uncertainties for their base variable
- **TUID extraction**: The unique identifier follows the format `YYYYMMDD-HHMMSS-fff-xxxxxx`

---

## Configuration

The Quantify backend requires a `QuantifyConfigData` configuration with the following fields:

| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `quantify_directory` | `Path` or `str` | Yes | Path to the Quantify data directory |
| `set_up` | `str` | Yes | Name of the experimental setup (added as `set-up` attribute) |
| `is_server_folder` | `bool` | No | Whether this is a server folder (default: `False`), e.g. on network drive of the university. |

### Example Configuration

Example using the `etiket-sdk` package:

```python
from etiket_sdk.sync import SyncSources

SyncSources.create(
    name="my_quantify_source",
    backend_identifier="etiket_sync_agent_quantify",
    config_data={
        "quantify_directory": "~/quantify-data",
        "set_up": "dilution_fridge_1",
        "is_server_folder": False
    },
    default_scope="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
)
```

## Live Sync

The Quantify backend supports **real-time synchronization** of running measurements.
We currenly do not support non-gridded datasets.

### How Live Detection Works

A dataset is considered "live" (still being written to) if:

1. **Recent modification**: The HDF5 file was modified within the last **2 minutes**
2. **Contains NaN values**: Quantify pre-allocates arrays with NaN values that get filled during measurement
3. **No newer datasets**: No other dataset directories have been created (indicating the measurement is still active)

### How Live Sync Works

The `XArrayReplicator` class performs real-time replication:

1. **Creates a qdrive dataset** with SWMR (Single-Writer Multiple-Reader) HDF5 mode
2. **Monitors file modifications** by checking the source file's modification time
3. **Tracks data using NaN cursor**: Follows the position of NaN values to detect new data
4. **Polls for updates** every **0.5 seconds**.
5. **Completes when**: All NaN values are replaced OR the 2-minute timeout is reached OR a newer dataset directory is created

### Timeout Behavior

| Condition | Timeout | Action |
|-----------|---------|--------|
| No file modification | 120 seconds (2 min) | Dataset marked as complete |
| Newer directory created | Immediate | Live sync stops, dataset marked complete |
| All NaN values filled | Immediate | Dataset marked complete |

### File Snapshot Safety

To avoid conflicts with concurrent file writes, the backend creates **temporary snapshots** of HDF5 files before reading:

```python
with with_dataset_snapshot(file_path) as safe_file:
    dataset = xr.open_dataset(safe_file, engine='h5netcdf')
    # Process dataset safely
```

This is necessary because Quantify continuously overwrites the HDF5 file during measurement. Creating a temporary copy avoids file corruption or read errors when the source file is being written to.

---

## Features

- **Direct Quantify integration**: Reads HDF5 datasets written by Quantify's data management system
- **Live sync support**: Real-time monitoring of running measurements via SWMR HDF5
- **Automatic TUID parsing**: Extracts timestamps and identifiers from Quantify's TUID format
- **Grid conversion**: Converts irregular datasets to gridded format for efficient visualization
- **Auxiliary file upload**: Syncs JSON, text, and other files alongside the main data

## Requirements

- Python >= 3.10
- xarray
- h5py
- h5netcdf

## License

Copyright © 2025 QHarbor. All Rights Reserved. See [LICENCE](LICENCE) for details.
