Metadata-Version: 2.1
Name: madsci.event_manager
Version: 0.7.1
Summary: The Modular Autonomous Discovery for Science (MADSci) Event and Logging Manager.
Author-Email: Tobias Ginsburg <tginsburg@anl.gov>, "Ryan D. Lewis" <ryan.lewis@anl.gov>, Casey Stone <cstone@anl.gov>, Doga Ozgulbas <dozgulbas@anl.gov>
License: MIT
Project-URL: Homepage, https://github.com/AD-SDL/MADSci
Requires-Python: >=3.10.0
Requires-Dist: madsci.common
Requires-Dist: madsci.client
Requires-Dist: pandas
Requires-Dist: matplotlib>=3.5.0
Requires-Dist: pytz
Description-Content-Type: text/markdown

# MADSci Event Manager

Handles distributed logging and events throughout a MADSci-powered Lab.

![MADSci Event Manager Architecture Diagram](./assets/event_manager.drawio.svg)

## Features

- Centralized logging from distributed lab components
- Event querying with structured filtering
- Arbitrary event data support with standard schema
- Python `logging`-style log levels
- Alert notifications (email, etc.)

## Installation

See the main [README](../../README.md#installation) for installation options. This package is available as:

- PyPI: `pip install madsci.event_manager`
- Docker: Included in `ghcr.io/ad-sdl/madsci`
- **Example configuration**: See [example_lab/managers/example_event.manager.yaml](../../examples/example_lab/managers/example_event.manager.yaml)

**Dependencies**: MongoDB database (see the [example_lab](../../examples/example_lab/))

## Usage

### Quick Start

Use the [example_lab](../../examples/example_lab/) as a starting point:

```bash
# Start with working example
docker compose up  # From repo root
# Event Manager available at http://localhost:8001/docs

# Or run standalone
python -m madsci.event_manager.event_server
```

### Manager Setup

For custom deployments, create an Event Manager definition:

```bash
madsci manager add -t event_manager
```

See [example_event.manager.yaml](../../examples/example_lab/managers/example_event.manager.yaml) for configuration options.

### Client

You can use MADSci's `EventClient` (`madsci.client.event_client.EventClient`) in your python code to log new events to the event manager, or fetch/query existing events.

```python
from madsci.client.event_client import EventClient
from madsci.common.types.event_types import Event, EventLogLevel, EventType

event_client = EventClient(
    name="my_component",
    event_server="http://localhost:8001", # Update with the host/port you configured for your EventManager server
)

# Modern structlog-style API (recommended)
event_client.info("This logs a simple string at the INFO level")
event_client.debug("Debug message with context", key="value", count=42)
event_client.warning("Warning with structured data", resource_id="res-123")
event_client.error("Error occurred", exc_info=True)  # Includes traceback

# With event types for structured querying
event_client.info(
    "Workflow completed",
    event_type=EventType.WORKFLOW_COMPLETE,
    workflow_id="wf-123",
    duration_ms=1500,
)

# Context binding for persistent metadata
event_client = event_client.bind(experiment_id="exp-456")
event_client.info("All subsequent logs include experiment_id")

# Legacy API (still supported)
event = Event(
    event_type=EventType.NODE_CREATE,
    log_level=EventLogLevel.DEBUG,
    event_data="This logs a NODE_CREATE event at the DEBUG level."
)
event_client.log(event)

# Get the 50 most recent events
event_client.get_events(number=50)
# Get all events from a specific node
event_client.query_events({"source": {"node_id": "01JJ4S0WNGEF5FQAZG5KDGJRBV"}})

event_client.alert(event) # Will force firing any configured alert notifiers on this event
```

### Logging with Context

Use the EventClient context system for hierarchical logging across components:

```python
from madsci.common.context import event_client_context, get_event_client

# Establish context at entry points
with event_client_context(name="my_experiment", experiment_id="exp-123") as logger:
    logger.info("Starting experiment")

    # All nested operations share context
    with event_client_context(name="workflow", workflow_id="wf-456") as wf_logger:
        wf_logger.info("Running workflow")  # Includes both experiment_id and workflow_id

# In library code, inherit context automatically
def utility_function():
    logger = get_event_client()  # Uses context if available
    logger.info("Utility running")
```

See the [Logging Guide](../../docs/guides/logging.md) for comprehensive documentation on structured logging and context management.

### OpenTelemetry Integration

The EventClient integrates with OpenTelemetry for distributed tracing:

```python
from madsci.client.event_client import EventClient, EventClientConfig

config = EventClientConfig(
    otel_enabled=True,
    otel_service_name="my_service",
    otel_exporter="otlp",
    otel_endpoint="http://localhost:4317",
)
client = EventClient(name="traced_component", config=config)

# Events automatically include trace_id and span_id
client.info("Traced event")  # Correlated with distributed traces
```

See the [Observability Guide](../../docs/guides/observability.md) for the full observability stack setup.

### Alerts

The Event Manager provides some native alerting functionality. A default alert level can be set in the event manager definition's `alert_level`, which will determine the minimum log level at which to send an alert. Calls directly to the `EventClient.alert` method will send alerts regardless of the `alert_level`.

You can configure Email Alerts by setting up an `EmailAlertsConfig` (`madsci.common.types.event_types.EmailAlertsConfig`) in the `email_alerts` field of your `EventManagerSettings`.

## Database Migration Tools

MADSci Event Manager includes automated MongoDB migration tools that handle schema changes and version tracking for the event management system.

### Features

- **Version Compatibility Checking**: Automatically detects mismatches between MADSci package version and MongoDB schema version
- **Automated Backup**: Creates MongoDB dumps using `mongodump` before applying migrations to enable rollback on failure
- **Schema Management**: Creates collections and indexes based on schema definitions
- **Index Management**: Ensures required indexes exist for optimal query performance
- **Location Independence**: Auto-detects schema files or accepts explicit paths
- **Safe Migration**: All changes are applied transactionally with automatic rollback on failure

### Usage

#### Standard Usage
```bash
# Run migration for events database (auto-detects schema file)
python -m madsci.common.mongodb_migration_tool --database madsci_events

# Migrate with explicit database URL
python -m madsci.common.mongodb_migration_tool --db-url mongodb://localhost:27017 --database madsci_events

# Use custom schema file
python -m madsci.common.mongodb_migration_tool --database madsci_events --schema-file /path/to/schema.json

# Create backup only
python -m madsci.common.mongodb_migration_tool --database madsci_events --backup-only

# Restore from backup
python -m madsci.common.mongodb_migration_tool --database madsci_events --restore-from /path/to/backup

# Check version compatibility without migrating
python -m madsci.common.mongodb_migration_tool --database madsci_events --check-version
```

#### Docker Usage
When running in Docker containers, use docker-compose to execute migration commands:

```bash
# Run migration for events database in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json'

# Create backup only in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json' --backup-only

# Check version compatibility in Docker
docker-compose run --rm event-manager python -m madsci.common.mongodb_migration_tool --db-url 'mongodb://mongodb:27017' --database 'madsci_events' --schema-file '/app/madsci/event_manager/schema.json' --check-version
```

### Server Integration

The Event Manager server automatically checks for version compatibility on startup. If a mismatch is detected, the server will refuse to start and display migration instructions:

```bash
DATABASE INITIALIZATION REQUIRED! SERVER STARTUP ABORTED!
The database exists but needs version tracking setup.
To resolve this issue, run the migration tool and restart the server.
```

### Schema File Location

The migration tool automatically searches for schema files in:
- `madsci/event_manager/schema.json`

### Backup Location

Backups are stored in `.madsci/mongodb/backups/` with timestamped filenames:
- Format: `madsci_events_backup_YYYYMMDD_HHMMSS`
- Can be restored using the `--restore-from` option

### Requirements

- MongoDB server running and accessible
- MongoDB tools (`mongodump`, `mongorestore`) installed
- Appropriate database permissions for the specified user
## API Reference

The Event Manager provides a REST API for logging and querying events. The API is available at `http://localhost:8001` by default.

### Event Operations

#### POST /event
Log a new event to the system.

**Request Body**: `Event` object
```json
{
  "event_type": "LOG_INFO",
  "log_level": 20,
  "event_data": "Event message or data",
  "alert": false
}
```

**Response**: The logged `Event` object with assigned `event_id` and `event_timestamp`.

#### GET /event/{event_id}
Retrieve a specific event by its ID.

**Parameters**:
- `event_id` (path): The unique event identifier

**Response**: `Event` object or 404 if not found.

#### GET /events
Get the latest events from the system.

**Query Parameters**:
- `number` (int, default: 100): Maximum number of events to return
- `level` (int, default: 0): Minimum log level to include

**Response**: Dictionary mapping event IDs to `Event` objects.

#### POST /events/query
Query events using MongoDB selector syntax.

**Request Body**: MongoDB query selector object
```json
{
  "source.node_id": "01JJ4S0WNGEF5FQAZG5KDGJRBV",
  "log_level": {"$gte": 20}
}
```

**Response**: Dictionary mapping event IDs to matching `Event` objects.

### Utilization Analysis

#### GET /utilization/sessions
Generate session-based utilization reports.

**Query Parameters**:
- `start_time` (string, optional): ISO format start time
- `end_time` (string, optional): ISO format end time
- `csv_format` (bool, default: false): Return data in CSV format
- `save_to_file` (bool, default: false): Save CSV to server filesystem
- `output_path` (string, optional): Server path to save CSV files

**Response**: JSON utilization report or CSV data.

### Standard Manager Endpoints

The Event Manager also provides standard manager endpoints:

- **GET /definition**: Get manager definition and metadata
- **GET /health**: Get health status including database connectivity and event counts

### Interactive API Documentation

Visit `http://localhost:8001/docs` when the Event Manager is running for interactive Swagger UI documentation.

## MongoDB Setup

The Event Manager requires MongoDB for event storage. For local development:

1. **Using Docker** (recommended):
   ```bash
   docker run -d -p 27017:27017 --name mongodb mongo:latest
   ```

2. **Using the example lab**:
   The [example_lab](../../examples/example_lab/) includes a pre-configured MongoDB instance.

3. **Configuration**:
   Set the database URL in your environment or manager configuration:
   ```yaml
   # In manager YAML config
   db_url: "mongodb://localhost:27017"
   collection_name: "madsci_events"
   ```

   Or as environment variables:
   ```bash
   export EVENT_DB_URL="mongodb://localhost:27017"
   export EVENT_COLLECTION_NAME="madsci_events"
   ```
