Metadata-Version: 2.4
Name: sysnet-sekm-migrate
Version: 0.2.0
Summary: SEKM migration utility: Elasticsearch → MongoDB/Beanie
Project-URL: Homepage, https://github.com/SYSNET-CZ/sekm-migrate
Project-URL: Documentation, https://github.com/SYSNET-CZ/sekm-migrate/blob/master/docs/MIGRATION_GUIDE.md
Project-URL: Repository, https://github.com/SYSNET-CZ/sekm-migrate
Project-URL: Bug Tracker, https://github.com/SYSNET-CZ/sekm-migrate/issues
License: AGPL-3.0-or-later
Requires-Python: >=3.12
Requires-Dist: elasticsearch[async]>=8.13
Requires-Dist: pydantic-settings>=2.3
Requires-Dist: python-dotenv>=1.0
Requires-Dist: sysnet-sekm-model
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
Requires-Dist: pytest>=8.2; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Description-Content-Type: text/markdown

# sysnet-sekm-migrate

[![PyPI](https://img.shields.io/pypi/v/sysnet-sekm-migrate)](https://pypi.org/project/sysnet-sekm-migrate/)
[![Python](https://img.shields.io/pypi/pyversions/sysnet-sekm-migrate)](https://pypi.org/project/sysnet-sekm-migrate/)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0)
[![GitHub](https://img.shields.io/badge/GitHub-SYSNET--CZ%2Fsekm--migrate-blue?logo=github)](https://github.com/SYSNET-CZ/sekm-migrate)

Migration utility for **SEKM** — transfers data from Elasticsearch indexes into
MongoDB documents managed by Beanie and `sysnet-docversion`.

Depends on [`sysnet-sekm-model`](https://pypi.org/project/sysnet-sekm-model/) for the
domain and persistence models.

## Installation

```bash
pip install sysnet-sekm-migrate
```

## Requirements

- Python ≥ 3.12
- Access to the source Elasticsearch cluster
- Access to a target MongoDB instance
- `.env` file with both ES and Mongo credentials (see below)

## Configuration

Create a `.env` file (or export environment variables):

```
ELK_URI=https://your-es-host:9200
ELK_USER=elastic
ELK_PASSWORD=...

MONGO_HOST=localhost
MONGO_PORT=27017
MONGO_USERNAME=...
MONGO_PASSWORD=...
MONGO_DB=sekm
```

## CLI usage

After installation the `sekm-migrate` command is available:

```bash
# Migrate everything (all 21 entities)
sekm-migrate

# Single entity
sekm-migrate --entity area-source

# Multiple entities
sekm-migrate --entity area-source --entity user

# Dry run — reads ES, validates, but writes nothing to MongoDB
sekm-migrate --entity area-source --dry-run

# Skip archive version history (faster)
sekm-migrate --entity as-object --no-history

# Limit documents (trial run)
sekm-migrate --entity sample --limit 10000

# List all 21 entity names
sekm-migrate --list-entities

# Verbose logging
sekm-migrate --log-level DEBUG
```

## Programmatic usage

```python
from sysnet_sekm_model.config import MongoSettings
from sysnet_sekm_model.db import init_mongo, close_mongo
from sysnet_sekm_migrate.services.elasticsearch import es_client
from sysnet_sekm_migrate.services.migration import migrate_all

settings = MongoSettings()
client = await init_mongo(settings.mongo_uri, settings.mongo_db)
try:
    async with es_client() as es:
        results = await migrate_all(
            es,
            entity_names=["area-source", "user"],  # None = all entities
            import_history=True,
            dry_run=False,
        )
finally:
    await close_mongo(client)

for entity, result in results.items():
    result.log_summary()
```

## Supported entities

| Entity | Collection | Versioned | ~Documents |
|---|---|---|---|
| `area-source` | `area_sources` | Yes | 29 000 |
| `as-object` | `as_objects` | Yes | 177 000 |
| `as-region` | `as_regions` | Yes | 2 800 |
| `as-document` | `as_documents` | Yes | 17 000 |
| `observation` | `observations` | Yes | 19 000 |
| `user` | `users` | Yes | 9 200 |
| `organisation` | `organisations` | Yes | 308 |
| `template` | `templates` | Yes | 6 500 |
| `dictionary` | `dictionaries` | Yes | 39 |
| `substance` | `substances` | Yes | 365 |
| `sample` | `samples` | No | ~15 300 000 |
| `sample-advanced` | `samples_advanced` | No | ~13 600 000 |
| `history` | `history_events` | No | 287 000 |
| `lock` | `lock_events` | No | 106 000 |
| `approval` | `approvals` | No | 302 |
| `region` | `regions` | No | 14 |
| `district` | `districts` | No | 77 |
| `municipality` | `municipalities` | No | 6 300 |
| `orp-district` | `orp_districts` | No | 206 |
| `pou-district` | `pou_districts` | No | 394 |
| `registry` | `registry_entries` | No | 13 000 |

Migration is idempotent — it can be run multiple times against the same MongoDB database.
Already-migrated documents are detected and skipped.

## Documentation

| Document | Contents |
|---|---|
| [Migration Guide](https://github.com/SYSNET-CZ/sekm-migrate/blob/master/docs/MIGRATION_GUIDE.md) | Full pipeline description, CLI reference, transformation details |
| [Data Model](https://github.com/SYSNET-CZ/sekm-migrate/blob/master/docs/DATA_MODEL.md) | Domain and persistence model reference |
| [Trial Migration Report](https://github.com/SYSNET-CZ/sekm-migrate/blob/master/docs/TRIAL_MIGRATION_REPORT.md) | Trial migration results and entity volumes |
| [Data Quality Report](https://github.com/SYSNET-CZ/sekm-migrate/blob/master/docs/DATA_QUALITY_REPORT.md) | Post-migration data quality findings and remediation |

## License

[AGPL-3.0](https://github.com/SYSNET-CZ/sekm-migrate/blob/master/LICENSE)
