---
title: Four-Layer Design
description: Detailed walkthrough of each architectural layer
---

Astromesh is organized into four distinct layers, each with a clear boundary and responsibility. This page walks through every layer in detail -- the modules it contains, the interfaces it exposes, and how it connects to adjacent layers.

For a high-level overview, see [Architecture Overview](/astromesh/architecture/overview/). For a request-level trace, see [Agent Execution Pipeline](/astromesh/architecture/agent-pipeline/).

## Layer 1: API Layer

**Module:** `astromesh/api/`

The API Layer is the entry point for all external communication. It accepts HTTP and WebSocket connections, validates input, and forwards requests to the Runtime Engine. It contains no business logic -- its only job is protocol translation.

### REST API

**Module:** `astromesh/api/main.py` + `astromesh/api/routes/`

The REST API is built with FastAPI and organized into route groups. Each route module exposes a `router` (FastAPI `APIRouter`) and a `set_runtime(runtime)` function that is called during bootstrap to inject the `AgentRuntime` instance.

| Route Group | Prefix | Key Endpoints | Description |
|-------------|--------|---------------|-------------|
| **Agents** | `/v1/agents` | `GET /`, `GET /{name}`, `POST /{name}/run` | List, inspect, and execute agents |
| **Memory** | `/v1/memory` | `GET /{agent}/history/{session}`, `DELETE /{agent}/history/{session}`, `GET /{agent}/semantic` | Query and manage conversation history and semantic memory |
| **Tools** | `/v1/tools` | `GET /`, `POST /execute` | List registered tools and execute them directly |
| **RAG** | `/v1/rag` | `POST /ingest`, `POST /query` | Ingest documents and query knowledge bases |
| **Health** | `/v1/health` | `GET /` | Health check and version information |

The main entry point (`main.py`) creates the FastAPI application, runs `AgentRuntime.bootstrap()` on startup, and calls `set_runtime()` on each route module so they can access the runtime without global state.

### WebSocket Streaming

**Module:** `astromesh/api/ws.py`

The WebSocket endpoint at `/v1/ws/agent/{agent_name}` provides real-time token streaming for agent responses. It uses a `ConnectionManager` that:

- Tracks active WebSocket connections per agent
- Accepts JSON messages with `query` and `session_id` fields
- Streams partial responses as tokens are generated by the LLM provider
- Handles disconnection and cleanup

Example client interaction:

```json
// Client sends:
{"query": "Explain quantum computing", "session_id": "session-42"}

// Server streams back token-by-token:
{"token": "Quantum", "done": false}
{"token": " computing", "done": false}
{"token": " uses", "done": false}
// ... more tokens ...
{"token": "", "done": true, "response": "Quantum computing uses..."}
```

### Channel Adapters

**Module:** `astromesh/channels/`

Channel adapters sit above the API Layer, bridging external messaging platforms to the Agent Runtime. They are configured in `config/channels.yaml` and each adapter handles:

- **Webhook verification** -- Platform-specific challenge/response (e.g., Meta's hub.verify_token)
- **Message parsing** -- Extracting text from platform-specific webhook payloads
- **Signature validation** -- HMAC verification of incoming webhooks
- **Response formatting** -- Converting agent output to platform-specific API calls
- **Background execution** -- Running agent execution in background tasks so webhooks respond within platform timeouts (e.g., Meta requires a response within 5 seconds)

Each channel maps to a `default_agent` that handles all conversations on that channel.

---

## Layer 2: Runtime Engine

**Module:** `astromesh/runtime/engine.py`

The Runtime Engine is the heart of Astromesh -- the control plane that turns declarative YAML configuration into running agent instances. It has two primary responsibilities: bootstrapping and execution.

### AgentRuntime Bootstrap

When the application starts, `AgentRuntime.bootstrap()` performs the following sequence:

```
config/agents/*.agent.yaml
        │
        ▼
┌─────────────────────────────────┐
│  1. Scan config/agents/ dir     │
│  2. Parse each .agent.yaml      │
│  3. Validate against schema     │
│  4. For each agent definition:  │
│     ├── Build ModelRouter       │──► Wire primary + fallback providers
│     ├── Build MemoryManager     │──► Configure backends + strategies
│     ├── Build ToolRegistry      │──► Register tools (internal, MCP, webhook, RAG)
│     ├── Select Pattern          │──► Instantiate orchestration pattern
│     ├── Build PromptEngine      │──► Register Jinja2 templates
│     ├── Build GuardrailsEngine  │──► Configure input/output rules
│     └── Create Agent instance   │──► Fully wired, ready to execute
└─────────────────────────────────┘
        │
        ▼
  agents: Dict[str, Agent]   ← Lookup table by agent name
```

The bootstrap process reads all YAML files, validates them, and assembles fully wired `Agent` objects. Each agent gets its own set of dependencies (router, memory, tools, etc.) based on its configuration.

### Agent Execution

When a request arrives for a specific agent, `AgentRuntime.run(agent_name, query, session_id)` looks up the agent by name and delegates to `Agent.run()`, which executes the full pipeline (covered in detail in [Agent Execution Pipeline](/astromesh/architecture/agent-pipeline/)).

Each `Agent` instance holds references to:

| Component | Class | Purpose |
|-----------|-------|---------|
| Model Router | `ModelRouter` | Select and call LLM providers with fallback |
| Memory Manager | `MemoryManager` | Build context from conversation, semantic, and episodic memory |
| Tool Registry | `ToolRegistry` | Discover, validate, and execute tools |
| Orchestration Pattern | `OrchestrationPattern` subclass | Control the reasoning loop (ReAct, PlanAndExecute, etc.) |
| Prompt Engine | `PromptEngine` | Render Jinja2 system prompts with variable injection |
| Guardrails Engine | `GuardrailsEngine` | Apply input/output safety rules |

---

## Layer 3: Core Services

Core Services are the domain-specific building blocks that agents use during execution. Each service has a well-defined interface and delegates to Layer 4 for concrete implementations.

### Model Router

**Module:** `astromesh/core/model_router.py`

The Model Router manages multi-provider LLM inference. Given a completion request, it ranks available providers using the configured routing strategy, checks circuit breaker state, and tries providers in order until one succeeds.

```
                    ┌─────────────┐
    Request ──────► │ ModelRouter │
                    │             │
                    │ 1. Rank     │──► Strategy-based ordering
                    │ 2. Try      │──► Circuit breaker check
                    │ 3. Fallback │──► Next provider on failure
                    └──────┬──────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
          ┌───────┐   ┌────────┐   ┌───────┐
          │Ollama │   │ OpenAI │   │ vLLM  │  ...
          └───────┘   └────────┘   └───────┘
```

#### Routing Strategies

| Strategy | Behavior | Best For |
|----------|----------|----------|
| `cost_optimized` | Cheapest provider first, based on `estimated_cost()` | Budget-sensitive workloads |
| `latency_optimized` | Fastest provider first, using exponential moving average of response times | Real-time applications |
| `quality_first` | Highest quality score first (configured per provider) | Tasks requiring maximum accuracy |
| `round_robin` | Rotate across providers evenly | Load distribution |
| `capability_match` | Filter by required capabilities (tool calling, vision), then apply secondary strategy | Tasks needing specific features |

#### Circuit Breaker

The circuit breaker protects the system from repeatedly calling failing providers:

- **Closed** (normal): Requests flow through. Failures are counted.
- **Open** (tripped): After **3 consecutive failures**, the circuit opens. All requests to that provider are immediately skipped for **60 seconds**.
- **Half-open** (testing): After the cooldown, one request is allowed through. If it succeeds, the circuit closes. If it fails, the circuit reopens.

When a provider's circuit is open, the router automatically tries the next provider in the ranked list. If all providers are tripped, the request fails with a clear error.

### Memory Manager

**Module:** `astromesh/core/memory.py`

The Memory Manager handles three distinct types of memory, each with pluggable backends. It provides two key operations: `build_context()` (assemble memory for prompt construction) and `persist_turn()` (store conversation turns after execution).

```
                    ┌──────────────────┐
                    │  MemoryManager   │
                    │                  │
                    │ build_context()  │──► Assemble context from all memory types
                    │ persist_turn()   │──► Store conversation turns
                    └──────┬───────────┘
                           │
           ┌───────────────┼───────────────┐
           ▼               ▼               ▼
    ┌──────────────┐ ┌───────────┐ ┌───────────┐
    │Conversational│ │ Semantic  │ │ Episodic  │
    │              │ │           │ │           │
    │Redis/PG/     │ │pgvector/  │ │PostgreSQL │
    │SQLite        │ │Chroma/    │ │           │
    │              │ │Qdrant/    │ │           │
    │              │ │FAISS      │ │           │
    └──────────────┘ └───────────┘ └───────────┘
```

#### Memory Types

| Type | Purpose | Backend Options |
|------|---------|----------------|
| **Conversational** | Chat history -- stores user and assistant messages for multi-turn conversations | Redis, PostgreSQL, SQLite |
| **Semantic** | Vector embeddings -- stores and retrieves information by semantic similarity | pgvector, ChromaDB, Qdrant, FAISS |
| **Episodic** | Event logs -- records significant events, tool calls, and outcomes for long-term learning | PostgreSQL |

#### Memory Strategies

Memory strategies control how conversational history is managed when it grows beyond what fits in the LLM's context window:

| Strategy | Description | Use Case |
|----------|-------------|----------|
| `sliding_window` | Keep the last N turns, discard older ones | Simple chatbots with short context needs |
| `summary` | Compress older turns into summaries using the LLM, keep recent turns verbatim | Long-running conversations that need historical awareness |
| `token_budget` | Fit as many recent turns as possible within a configured token limit | Maximizing context usage without exceeding model limits |

### Tool Registry

**Module:** `astromesh/core/tools.py`

The Tool Registry is the central authority for all tools an agent can use. It handles tool discovery, registration, permission checking, rate limiting, and schema generation for LLM function calling.

#### Tool Types

| Type | Source | Description |
|------|--------|-------------|
| `internal` | Python functions | Registered directly in code. Fastest execution path. |
| `mcp` | MCP servers | Discovered from external MCP servers via stdio, SSE, or HTTP transport. Tools are fetched at startup and cached. |
| `webhook` | HTTP endpoints | External REST APIs called via HTTP. Supports custom headers, authentication, and payload mapping. |
| `rag` | RAG pipeline | Exposes a RAG pipeline as a callable tool, allowing agents to search knowledge bases during reasoning. |

#### Features

- **Schema generation** -- Automatically generates JSON Schema for each tool, compatible with the function calling format expected by LLM providers (OpenAI-style `tools` parameter).
- **Permission filtering** -- Each agent definition specifies which tools it has access to. The registry filters available tools per agent at execution time.
- **Rate limiting** -- Tools can have per-agent rate limits to prevent runaway tool calls during orchestration loops.

### Prompt Engine

**Module:** `astromesh/core/prompt_engine.py`

The Prompt Engine renders Jinja2 templates into system prompts for LLM calls. Key features:

- **SilentUndefined** -- Missing template variables render as empty strings instead of raising errors. This allows prompts to gracefully handle optional context (e.g., no semantic memory results).
- **Template registration** -- Templates are registered during agent bootstrap from the `prompts.system` field in agent YAML.
- **Variable injection** -- At render time, the engine receives variables from the Memory Manager (conversation history, semantic results, episodic events) and injects them into the template.

Example system prompt template:

```jinja2
You are {{ agent_name }}, a {{ description }}.

{% if conversation_history %}
Previous conversation:
{% for msg in conversation_history %}
{{ msg.role }}: {{ msg.content }}
{% endfor %}
{% endif %}

{% if semantic_context %}
Relevant knowledge:
{{ semantic_context }}
{% endif %}
```

### Guardrails Engine

**Module:** `astromesh/core/guardrails.py`

The Guardrails Engine applies safety checks on both input (before the agent processes a query) and output (before the response is returned to the caller).

| Guardrail | Direction | Description |
|-----------|-----------|-------------|
| `pii_detection` | Input & Output | Detects and redacts emails, phone numbers, SSNs, and credit card numbers using regex patterns |
| `topic_filter` | Input | Blocks messages matching forbidden topics defined in configuration |
| `max_length` | Input | Enforces maximum character limits on incoming queries |
| `cost_limit` | Output | Enforces token-per-turn limits to prevent runaway LLM usage |
| `content_filter` | Output | Blocks responses containing forbidden keywords or patterns |

Guardrails run in the order they are defined in the agent's YAML configuration. Each guardrail can either **pass** (allow the message through), **redact** (modify the message and continue), or **block** (reject the message with an error).

---

## Layer 4: Infrastructure

Infrastructure contains all concrete implementations of the interfaces defined in Layer 3. These are the adapters, drivers, and backends that do the actual work.

### LLM Providers

**Module:** `astromesh/providers/`

All providers implement `ProviderProtocol`, a `runtime_checkable` Python Protocol defined in `astromesh/providers/base.py`.

#### ProviderProtocol Methods

| Method | Return Type | Description |
|--------|-------------|-------------|
| `complete(messages, **kwargs)` | `CompletionResult` | Synchronous (non-streaming) completion |
| `stream(messages, **kwargs)` | `AsyncIterator[StreamChunk]` | Streaming completion, yielding tokens |
| `health_check()` | `bool` | Check if the provider is reachable |
| `supports_tools()` | `bool` | Whether the provider supports function calling |
| `supports_vision()` | `bool` | Whether the provider supports image inputs |
| `estimated_cost(tokens)` | `float` | Estimated cost for a given token count |
| `avg_latency_ms` | `float` | Exponential moving average of response latency |

#### Provider Implementations

| Provider | Backend | Endpoint Style | Notes |
|----------|---------|---------------|-------|
| `OllamaProvider` | Ollama | `/api/chat` | Native Ollama API format |
| `OpenAICompatProvider` | OpenAI API | `/v1/chat/completions` | Works with any OpenAI-compatible API |
| `VLLMProvider` | vLLM | OpenAI-compatible | High-throughput serving with PagedAttention |
| `LlamaCppProvider` | llama.cpp | OpenAI-compatible | CPU/GPU inference with GGUF models |
| `HFTGIProvider` | HuggingFace TGI | OpenAI-compatible | Text Generation Inference server |
| `ONNXProvider` | ONNX Runtime | Local inference | In-process inference, no network calls |

Each provider tracks its own latency history and cost estimates, which the Model Router uses for routing decisions.

### Orchestration Patterns

**Module:** `astromesh/orchestration/`

Orchestration patterns control how agents reason and use tools. Each pattern implements the `OrchestrationPattern` abstract base class with an `execute()` method that runs the reasoning loop.

| Pattern | Module | Description | Typical Use Case |
|---------|--------|-------------|-----------------|
| `ReAct` | `patterns.py` | Think, Act, Observe loop. The LLM decides when to call tools and when to produce a final answer. Iterates up to `max_iterations`. | General-purpose agents that need tools |
| `PlanAndExecute` | `patterns.py` | First, the LLM creates a plan (list of steps). Then, each step is executed sequentially. | Multi-step tasks with clear decomposition |
| `ParallelFanOut` | `patterns.py` | Send the same query to multiple sub-models simultaneously, merge results. | Ensemble approaches, multi-perspective analysis |
| `Pipeline` | `patterns.py` | Chain multiple processing steps sequentially, where each step's output feeds the next. | Data transformation, multi-stage processing |
| `Supervisor` | `supervisor.py` | A supervisor agent delegates sub-tasks to worker agents and combines their results. | Complex tasks requiring specialized sub-agents |
| `Swarm` | `swarm.py` | Agents hand off conversations to each other based on context. No central coordinator. | Customer service routing, multi-domain conversations |

### RAG Pipeline

**Module:** `astromesh/rag/`

The RAG (Retrieval-Augmented Generation) pipeline connects document ingestion to knowledge-augmented agent responses.

```
Ingestion Flow:
Documents → Chunking → Embedding → Vector Store

Query Flow:
Query → Embedding → Vector Search → Reranking → Top-K results
```

Each stage is pluggable:

| Stage | Options | Module |
|-------|---------|--------|
| **Chunking** | Fixed-size, Recursive (split on separators), Sentence-aware, Semantic (embedding-based boundaries) | `astromesh/rag/chunking/` |
| **Embeddings** | HuggingFace Inference API, SentenceTransformers (local), Ollama | `astromesh/rag/embeddings/` |
| **Vector Store** | pgvector, ChromaDB, Qdrant, FAISS | `astromesh/rag/stores/` |
| **Reranking** | Cross-encoder (local), Cohere Rerank API | `astromesh/rag/reranking/` |

RAG pipelines are configured in `config/rag/*.rag.yaml` with `apiVersion: astromesh/v1, kind: RAGPipeline`.

### MCP Integration

**Module:** `astromesh/mcp/`

Astromesh supports the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) in both directions:

#### MCP Client (`astromesh/mcp/client.py`)

Connects to external MCP servers to discover and invoke remote tools. Supports three transport mechanisms:

| Transport | Protocol | Use Case |
|-----------|----------|----------|
| **stdio** | stdin/stdout JSON-RPC | Local MCP servers, CLI tools |
| **SSE** | Server-Sent Events over HTTP | Remote MCP servers with streaming |
| **HTTP** | Standard HTTP JSON-RPC | Remote MCP servers, simple request/response |

At startup, the MCP client connects to configured servers, fetches their tool manifests, and registers discovered tools in the Tool Registry. Tools are then available to agents just like internal tools.

#### MCP Server (`astromesh/mcp/server.py`)

Exposes Astromesh agents as MCP tools via a JSON-RPC endpoint at `/mcp`. This allows other MCP-compatible systems to call your Astromesh agents as if they were tools.

### ML Model Registry

**Module:** `astromesh/ml/`

The ML Model Registry manages non-LLM machine learning models for tasks like classification, embedding generation, and custom inference:

- **Registry** (`model_registry.py`) -- Register, version, load, and serve ML models
- **Serving** (`serving/`) -- ONNX Runtime and PyTorch model servers for inference
- **Training** (`training/`) -- Classifier and embedding fine-tuning pipelines

### Observability

**Module:** `astromesh/observability/`

The observability stack provides three pillars of monitoring:

```
Agent Execution ──► TelemetryManager ──► OpenTelemetry Collector ──► Jaeger/Zipkin
                         │
                    MetricsCollector ──► Prometheus ──► Grafana
                         │
                    CostTracker ──► Usage records + budget alerts
```

| Component | Module | Description |
|-----------|--------|-------------|
| **TelemetryManager** | `telemetry.py` | Distributed tracing with OpenTelemetry. Creates spans for each pipeline step (guardrails, memory, routing, tool calls). Falls back to `_NoOpSpan` when OpenTelemetry is not installed. |
| **MetricsCollector** | `metrics.py` | Prometheus metrics: request counts (by agent, status), latency histograms (by agent, provider), active agents gauge. |
| **CostTracker** | `cost_tracker.py` | Per-provider usage records tracking token counts and estimated costs. Supports budget enforcement (alert or block when budget is exceeded) and grouped cost reports. |

---

## Layer Interaction Summary

The following rules govern how layers interact:

1. **Layer 1 (API)** calls only **Layer 2 (Runtime)**. API routes call `runtime.run()` and nothing else.
2. **Layer 2 (Runtime)** calls only **Layer 3 (Core Services)**. The Agent orchestrates calls to ModelRouter, MemoryManager, ToolRegistry, PromptEngine, and GuardrailsEngine.
3. **Layer 3 (Core Services)** calls only **Layer 4 (Infrastructure)**. The ModelRouter calls providers. The MemoryManager calls memory backends. The ToolRegistry calls tool implementations.
4. **Layer 4 (Infrastructure)** calls only external systems (LLM endpoints, databases, vector stores, APIs).

No layer ever skips a level. The API Layer never directly calls a provider. The Runtime Engine never directly accesses a database. This strict layering makes the system testable (each layer can be tested in isolation with mocked dependencies) and maintainable (changes to infrastructure never ripple up to the API).
