Metadata-Version: 2.4
Name: etiket_sync_agent
Version: 0.3.0b1
Summary: Sync agent for managing eTiKeT data synchronization from various backend sources
Author: QHarbor team
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://qharbor.nl
Project-URL: Documentation, https://docs.qharbor.nl
Keywords: etiket,sync,data-synchronization,scientific-data,qcodes,quantify,labber
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft :: Windows
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENCE
Requires-Dist: etiket_client>=0.3.0b1
Requires-Dist: etiket-sync-agent-native
Requires-Dist: etiket-sync-agent-folderbase
Requires-Dist: sqlalchemy<3.0,>=2.0
Requires-Dist: greenlet>=3.0
Requires-Dist: aiosqlite>=0.19
Requires-Dist: alembic<2.0,>=1.10
Requires-Dist: platformdirs<5.0,>=3.0
Requires-Dist: watchdog>=3.0
Requires-Dist: xarray>=2024.1.0
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23; extra == "test"
Requires-Dist: aiosqlite>=0.19; extra == "test"
Requires-Dist: truststore; extra == "test"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: aiosqlite>=0.19; extra == "dev"
Requires-Dist: truststore; extra == "dev"
Dynamic: license-file

# eTiKeT Sync Agent

A Python library for synchronizing scientific data from various backend sources to the eTiKeT platform.

## Overview

The eTiKeT Sync Agent provides a unified interface for automatically discovering and synchronizing experimental datasets from different data acquisition systems. It supports both database-backed sources (e.g., QCoDeS, Core-tools) and file-based sources (e.g., Quantify, Labber, generic folder structures).

### Key Features

- **Plugin-based backend system** — Easily extensible architecture using Python entry points
- **Multiple data source support** — Native eTiKeT, QCodes, Quantify, Labber, Core-tools, and generic folder-based sources
- **Live dataset synchronization** — Real-time sync for ongoing measurements
- **File converters** — Transform data formats during synchronization
- **Persistent tracking** — SQLite database for tracking the sync state of each dataset
- **Manifest-based file monitoring** — Efficient change detection for file-based sources

### Supported Backends

| Backend | Type | Description |
|---------|------|-------------|
| `etiket_sync_agent_native` | Native | eTiKeT native datasets |
| `etiket_sync_agent_folderbase` | File-based | Generic folder structures |
| `etiket_sync_agent_qcodes` | Database | QCoDeS measurement databases |
| `etiket_sync_agent_quantify` | File-based | Quantify scheduler data |
| `etiket_sync_agent_labber` | File-based | Labber measurement files |
| `etiket_sync_agent_core_tools` | Database | Core-tools measurements |

## Installation

We recommend using the **qdrive** package to install and manage the sync agent as a background service. See the [QHarbor documentation](https://docs.qharbor.nl) for setup instructions.

For standalone installation:

```bash
pip install etiket_sync_agent
```

Install backends as needed:

```bash
pip install etiket_sync_agent_qcodes
pip install etiket_sync_agent_quantify
```

## Usage

The sync agent is typically run as a background service managed by the **qdrive** package. Once configured, it continuously monitors your data sources and synchronizes new datasets to eTiKeT.

### Running Manually

To run the sync agent manually:

```bash
python -m etiket_sync_agent
```

For more control (e.g., enabling debug logging), you can run the sync loop directly:

```python
import logging
import sys

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    handlers=[logging.StreamHandler(sys.stdout)]
)

from etiket_sync_agent.run import sync_loop

# Run sync loop: 0 = run indefinitely, N = run N iterations
sync_loop(0)
```

## CLI Tools

The package provides CLI tools to scaffold new backends and converters. These scaffolds give you a ready-to-use package structure with all the boilerplate code in place—just fill in your implementation logic.

### How It Works

The generated packages include pre-configured entry points in their `pyproject.toml`. Once installed, the sync agent automatically discovers and registers these backends or converters via Python's [entry points mechanism](https://packaging.python.org/en/latest/specifications/entry-points/).

> **Tip:** For working examples, check out the existing backend packages like [`etiket-sync-agent-qcodes`](https://pypi.org/project/etiket-sync-agent-qcodes/) on PyPI.
>
> **Need help?** Contact us at [support@qharbor.nl](mailto:support@qharbor.nl).

---

### Generate Backend Scaffold

Creates a new sync agent backend package for synchronizing data from a specific source (e.g., a database, file system, or instrument).

```bash
generate_sync_agent_scaffold <backend> [options]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `backend` | Identifier for your backend (e.g., `qcodes`, `labber`, `my-instrument`) |

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--version` | `0.0.1` | Initial package version |
| `--dependency` | — | Add a dependency (repeatable). You can also edit `pyproject.toml` manually. |
| `--base` | `database` | Base class template: `database` or `file` (see below) |
| `--default-scope` | `REQUIRED` | Scope assignment policy (see below) |
| `--live-sync` | `False` | Set to `True` if you plan to implement live synchronization |
| `--level` | `-1` | *File-based only.* Directory level for dataset detection (see below) |

**Understanding `--base`:**

- **`database`** — Use when syncing from a database (e.g., SQLite from QCoDeS). The sync agent queries the database for new datasets.
- **`file`** — Use when syncing folder-based datasets where each folder or file represents a dataset.

**Understanding `--default-scope`:**

- **`REQUIRED`** — Users must specify the target scope (project/folder) for each sync source.
- **`OPTIONAL`** — A default scope can be inferred, but users may override it.
- **`DISABLED`** — The backend determines the scope automatically; users cannot change it.

**Understanding `--level` (file-based only):**

- **`-1`** — Every folder containing a `_QH_dataset_info.yaml` file becomes a dataset.
- **`1`** — Each immediate subfolder of the root is a dataset.
- **`2+`** — Folders at that depth level become datasets.

**Example — Database Backend:**

```bash
# Create a backend for a custom SQLite database
generate_sync_agent_scaffold my-backend \
    --base database \
    --dependency "sqlalchemy>=2.0" \
    --default-scope REQUIRED
```

This generates:

```
etiket-sync-agent-my-backend/
├── pyproject.toml                      # Package config with entry points
└── etiket_sync_agent_my_backend/
    ├── __init__.py                     # Exports sync and config classes
    ├── my_backend_config_class.py   # Configuration dataclass
    └── my_backend_sync_class.py     # Sync logic (implement this)
```

**Example — File-based Backend:**

```bash
# Create a backend for experiment folders at depth level 2
generate_sync_agent_scaffold lab-data \
    --base file \
    --level 2 \
    --live-sync
```

---

### Generate Converter Scaffold

Creates a new file converter package for transforming file formats during synchronization (e.g., CSV → HDF5). Converters are used by default in the folderbase backend, but you can also use them in your own custom backends.

```bash
generate_converter_scaffold <converter> [options]
```

**Arguments:**

| Argument | Description |
|----------|-------------|
| `converter` | Identifier for your converter (e.g., `csv-to-hdf5`, `mat-to-zarr`) |

**Options:**

| Option | Default | Description |
|--------|---------|-------------|
| `--version` | `0.0.1` | Initial package version |
| `--dependency` | — | Add a dependency (repeatable) |
| `--entry-name` | — | Custom entry point name (defaults to sanitized converter name) |

**Example:**

```bash
# Create a converter for CSV to HDF5
generate_converter_scaffold csv-to-hdf5 \
    --dependency h5py \
    --dependency pandas
```

This generates:

```
etiket-sync-agent-csv-to-hdf5-converters/
├── pyproject.toml                           # Package config with entry points
└── etiket_sync_agent_csv_to_hdf5_converters/
    ├── __init__.py                          # Exports converter class
    └── csv_to_hdf5_converter.py             # Converter logic (implement this)
```

After implementing your converter, install it and the sync agent will automatically use it for matching file types.

## License

See [LICENCE](LICENCE) for details.
