# ADR-003: Saga Pattern

**Status:** Accepted
**Date:** 2026-04-17
**Authors:** MCP Hangar Team

## Context

MCP Hangar needs to manage complex, long-running processes that span multiple domain aggregates or even external services (e.g., K8s resources). These operations, such as MCP server failover or recovery, cannot be handled as a single atomic transaction. Traditional distributed transactions (2PC) are not scalable and add significant complexity to a system where individual components may fail or restart independently.

## Decision

We have implemented the Saga pattern to manage distributed, multi-step business processes with compensating actions for failure handling.

### Saga Types

We support three primary saga types:

1. **MCP Server Failover Saga** (`mcp_hangar.application.sagas.mcp_server_failover_saga.ProviderFailoverSaga`): Orchestrates primary-to-backup MCP server transitions in three steps (start backup, await primary, failback).
2. **MCP Server Recovery Saga** (`mcp_hangar.application.sagas.mcp_server_recovery_saga.ProviderRecoverySaga`): Automatically restarts degraded MCP servers with backoff.
3. **Group Rebalance Saga** (`mcp_hangar.application.sagas.group_rebalance_saga.GroupRebalanceSaga`): Manages load balancing and re-distribution of MCP servers within a group.

### Implementation Details

- **SagaManager**: Orchestrates saga execution and persists state (`mcp_hangar.infrastructure.saga_manager.SagaManager`).
- **Step-based Sagas**: Sagas define a series of named steps. Each step includes a command and a corresponding compensation command to roll back changes if a subsequent step fails.
- **Event-triggered Sagas**: `EventTriggeredSaga` instances (`mcp_hangar.application.ports.saga.EventTriggeredSaga`) listen for domain events and initiate step-based sagas or other commands.
- **Compensation**: If a step in a saga fails, the `SagaManager` executes the compensation commands for all previously successful steps in reverse order, ensuring the system returns to a consistent state.
- **Scheduled Commands**: Sagas can schedule future commands with specific delays, which is used for auto-failback logic (`sm.schedule_command`).

## Consequences

### Positive

- **Fault Tolerance**: Sagas provide a robust framework for handling failures in long-running processes through automated compensations.
- **Event-driven Orchestration**: Sagas naturally integrate with our domain event system, allowing complex workflows to react to system state changes.
- **Visibility**: Saga state is persisted and observable, providing a clear view of the current progress of long-running operations.
- **Maintainability**: Complex logic is encapsulated into discrete, testable steps rather than being scattered across multiple event handlers.

### Negative

- **Eventual Consistency**: Sagas do not provide ACID guarantees; the system may be in an intermediate state while a saga is in progress.
- **Design Complexity**: Defining effective compensation logic can be challenging, especially for external side effects that are difficult to undo.
- **State Management**: Saga state must be carefully managed and persisted across application restarts to ensure that long-running processes can resume correctly.
