Metadata-Version: 2.4
Name: contextforge-apiconnect-fedapimgmt
Version: 1.0.0
Summary: IBM API Connect Federated API Management Plugin for ContextForge - automated MCP server and tool synchronization with centralized governance
Author-email: APIC Federated API Management Team <apic-fedapimgmt-team-dg@ibm.com>
Maintainer-email: APIC Federated API Management Team <apic-fedapimgmt-team-dg@ibm.com>
License: Apache-2.0
Keywords: contextforge,mcp,model-context-protocol,api-connect,federated-api-management,plugin,ibm,governance,monitoring,synchronization,MCP,API,gateway,proxy,tools,model context protocol,federation,security,authentication,ai control plane,control plane,governance,observability
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: System :: Monitoring
Classifier: Topic :: System :: Systems Administration
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx>=0.24.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: cpex>=0.1.0
Requires-Dist: mcp<2.0.0,>=1.28.1
Requires-Dist: mcp-contextforge-gateway>=1.0.6
Provides-Extra: dev
Requires-Dist: pytest>=7.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
Requires-Dist: black>=23.0.0; extra == "dev"
Requires-Dist: ruff>=0.1.0; extra == "dev"
Requires-Dist: mypy>=1.0.0; extra == "dev"
Provides-Extra: mock-fedapimgmt
Requires-Dist: fastapi>=0.109.0; extra == "mock-fedapimgmt"
Requires-Dist: uvicorn[standard]>=0.27.0; extra == "mock-fedapimgmt"
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "test"
Requires-Dist: httpx>=0.24.0; extra == "test"
Requires-Dist: fastapi>=0.109.0; extra == "test"
Requires-Dist: uvicorn[standard]>=0.27.0; extra == "test"
Provides-Extra: integration-test
Requires-Dist: mcp<2.0.0,>=1.28.1; extra == "integration-test"
Requires-Dist: mcp-contextforge-gateway>=1.0.6; extra == "integration-test"
Requires-Dist: pytest>=7.0.0; extra == "integration-test"
Requires-Dist: pytest-asyncio>=0.21.0; extra == "integration-test"
Dynamic: license-file

# ContextForge APIConnect Federated API Management Plugin

[![PyPI version](https://badge.fury.io/py/contextforge-apiconnect-fedapimgmt.svg)](https://badge.fury.io/py/contextforge-apiconnect-fedapimgmt)
[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

**IBM API Connect Federated API Management integration plugin for [MCP ContextForge](https://github.com/IBM/mcp-context-forge).** Enables centralized governance, discovery, and real-time monitoring of your MCP infrastructure by automatically synchronizing servers, tools, and runtime health data with IBM API Connect's Federated API Management platform.

---

## Table of Contents

- [ContextForge APIConnect Federated API Management Plugin](#contextforge-apiconnect-federated-api-management-plugin)
  - [Table of Contents](#table-of-contents)
  - [Overview](#overview)
  - [Terminology](#terminology)
  - [Prerequisites](#prerequisites)
  - [Installation](#installation)
  - [Deployment Guide](#deployment-guide)
    - [Local Deployment](#local-deployment)
    - [Docker Deployment](#docker-deployment)
    - [Docker Compose Deployment](#docker-compose-deployment)
    - [Kubernetes and Helm Deployment](#kubernetes-and-helm-deployment)
  - [Configuration Reference](#configuration-reference)
    - [Plugin Registration](#plugin-registration)
    - [Required Parameters](#required-parameters)
    - [Authentication Parameters](#authentication-parameters)
    - [Runtime Metadata Parameters](#runtime-metadata-parameters)
    - [Synchronization Parameters](#synchronization-parameters)
    - [TLS/SSL Parameters](#tlsssl-parameters)
  - [Authentication](#authentication)
    - [self-hosted (Basic Auth)](#self-hosted-basic-auth)
    - [v12 (Bearer Token)](#v12-bearer-token)
    - [IW (IBM Watson Platform Token)](#iw-ibm-watson-platform-token)
  - [TLS/SSL Configuration](#tlsssl-configuration)
    - [Default — System CA bundle](#default--system-ca-bundle)
    - [Custom CA Certificate](#custom-ca-certificate)
    - [Mutual TLS (mTLS)](#mutual-tls-mtls)
  - [How It Works](#how-it-works)
    - [Activity Overview](#activity-overview)
    - [Circuit Breaker](#circuit-breaker)
  - [Troubleshooting](#troubleshooting)
    - [Plugin not loading](#plugin-not-loading)
    - [Runtime registration fails](#runtime-registration-fails)
    - [Authentication fails (401 Unauthorized)](#authentication-fails-401-unauthorized)
    - [TLS certificate verification fails](#tls-certificate-verification-fails)
    - [Mutual TLS handshake fails](#mutual-tls-handshake-fails)
    - [Virtual Servers (MCP Servers) not appearing in IBM APIConnect Federated API Management](#virtual-servers-mcp-servers-not-appearing-in-ibm-apiconnect-federated-api-management)
    - [Tools not appearing in IBM APIConnect Federated API Management](#tools-not-appearing-in-ibm-apiconnect-federated-api-management)
    - [Circuit breaker is open](#circuit-breaker-is-open)
    - [Enable debug logging](#enable-debug-logging)
    - [Collecting diagnostic information](#collecting-diagnostic-information)
  - [Security Best Practices](#security-best-practices)
  - [License](#license)
  - [Support](#support)

---

## Overview

The ContextForge APIConnect Federated API Management Plugin connects your MCP ContextForge Gateway to **IBM API Connect Federated API Management**. Once installed, it runs in the background and continuously:

- **Registers** the ContextForge gateway instance as a **Runtime** in IBM API Connect Federated API Management
- **Synchronizes** all Virtual Servers (MCP Servers) and their Tools to the Federated API Management Asset Catalog
- **Sends heartbeats** to indicate that the runtime is alive and healthy
- **Collects and reports metrics** (request counts, error rates, latencies) to Federated API Management

This gives operators a single control plane in IBM API Connect to discover, govern, and monitor all MCP resources across any number of distributed ContextForge deployments.

---

## Terminology

Understanding the relationship between MCP ContextForge concepts and IBM API Connect Federated API Management concepts is essential for correct configuration.

| MCP ContextForge Term | IBM APIConnect Federated API Management Term | Description |
|---|---|---|
| **Gateway** / **MCP Gateway** | **Runtime** | A running ContextForge instance. Each gateway registers itself as a Runtime in IBM APIConnect Federated API Management with a unique `runtime_id`. |
| **Virtual Server** | **MCP Server** | A ContextForge Virtual Server (a REST/gRPC API wrapped as an MCP server) is represented as an MCP Server in the IBM APIConnect Federated API Management Asset Catalog. |
| **Tool** | **MCP Tool** | A function exposed by a Virtual Server that AI agents can invoke. Tools are registered under their parent MCP Server in IBM APIConnect Federated API Management. |


A single IBM API Connect Federated API Management installation can govern **multiple Runtimes** — one per ContextForge deployment (e.g., production, staging, regional gateways). Each Runtime independently synchronizes its own set of Virtual Servers and Tools.

---

## Prerequisites

Before you install the plugin, ensure that your environment meets the following requirements:

 | Component | Minimum version / Requirement | Description |
 | --- | --- | --- |
 |Python | 3.11 or later but less than 3.14 | Required when running MCP ContextForge in a local environment. |
 | ContextForge MCP Gateway | 1.0.6 or later | Requires a version that supports the cpex plugin framework. |
 |IBM API Connect Federated API Management (SaaS) | 12.1.1.0 or later | Minimum supported version for SaaS deployments. The instance must be accessible over HTTPS from the ContextForge host. |
 |IBM API Connect Federated API Management (On-Premises) | 12.1.1.2 or later| Minimum supported version for on-premises deployments. The instance must be accessible over HTTPS from the ContextForge host.|

If you are using an earlier version of IBM API Connect Federated API Management, upgrade to a supported release before enabling the plugin. Earlier versions do not provide the API endpoints required by the plugin, which can result in 404 Not Found or 501 Not Implemented errors during runtime registration and synchronization.

---

## Installation

Install the plugin from PyPI into the same Python environment where ContextForge is running:

```bash
pip install contextforge-apiconnect-fedapimgmt
```

To pin a specific version:

```bash
pip install contextforge-apiconnect-fedapimgmt==1.0.0
```

Verify the installation:

```bash
pip show contextforge-apiconnect-fedapimgmt
```

---

## Deployment Guide

The plugin runs inside MCP ContextForge. The deployment steps differ slightly depending on how you run ContextForge.

### Local Deployment

Use this approach when running MCP ContextForge directly on your machine or a bare-metal/VM server.

**Step 1 — Install the plugin**

```bash
# Activate the same virtual environment used by ContextForge
source ~/.venv/mcpgateway/bin/activate

pip install contextforge-apiconnect-fedapimgmt
```

**Step 2 — Create the plugin configuration file**

Create `plugins/config.yaml` in your ContextForge working directory:

```yaml
plugins:
  - name: "APIConnectFAM"
    kind: "contextforge_apiconnect_fam.apiconnect_fam.APIConnectFAMPlugin"
    description: "IBM API Connect FAM integration"
    version: "1.0.0"
    author: "Your Name"
    hooks: []
    tags: ["fedapimgmt", "sync", "monitoring"]
    mode: "permissive"
    priority: 1000
    config:
      apiconnect_fedapimgmt_enabled: true
      apiconnect_fedapimgmt_base_url: "https://fedapimgmt.example.com"
      apiconnect_fedapimgmt_runtime_id: "prod-gateway-01"

      # Authentication — choose one instance type
      apiconnect_fedapimgmt_instance_type: "v12"
      apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"
      apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"

      apiconnect_fedapimgmt_verify_ssl: true
      apiconnect_fedapimgmt_runtime_name: "Production Gateway"
      apiconnect_fedapimgmt_runtime_deployment_type: "ON_PREMISE"
```

**Step 3 — Enable plugins in your ContextForge `.env`**

```bash
PLUGINS_ENABLED=true
PLUGINS_CONFIG_FILE=plugins/config.yaml
```

**Step 4 — Set credentials as environment variables**

```bash
export APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER="your-client-id"
export APICONNECT_FEDAPIMGMT_AUTH_SECRET="your-api-key"
```

**Step 5 — Start ContextForge**

```bash
make dev      # development mode
# or
make serve    # production mode
```

**Step 6 — Verify**

```bash
tail -f logs/mcpgateway.log | grep -i "apiconnect\|fedapimgmt\|runtime"
```

Expected output:
```
INFO: Loading plugin: APIConnectFedAPIMgmt
INFO: Initializing APIConnectFedAPIMgmtPlugin with interval=60s
INFO: Runtime registered with IBM APIConnect Federated API Management: prod-gateway-01
INFO: IBM APIConnect Federated API Management heartbeat sent successfully
```

---

### Docker Deployment

Use this approach when running ContextForge as a standalone Docker container.

**Step 1 — Create a custom Dockerfile**

Create `Dockerfile.fedapimgmt` alongside your ContextForge Dockerfile:

```dockerfile
FROM ghcr.io/ibm/mcp-context-forge:latest

# Install the IBM APIConnect Federated API Management plugin
RUN pip install --no-cache-dir contextforge-apiconnect-fedapimgmt

# Copy plugin configuration
COPY plugins/config.yaml /app/plugins/config.yaml
```

**Step 2 — Create `plugins/config.yaml`**

```yaml
plugins:
  - name: "APIConnectFedAPIMgmt"
    kind: "contextforge_apiconnect_fedapimgmt.apiconnect_fedapimgmt.APIConnectFedAPIMgmtPlugin"
    description: "IBM APIConnect Federated API Management integration"
    version: "1.0.0"
    author: "Your Name"
    hooks: []
    tags: ["fedapimgmt", "sync", "monitoring"]
    mode: "permissive"
    priority: 1000
    config:
      apiconnect_fedapimgmt_enabled: true
      apiconnect_fedapimgmt_base_url: "https://fedapimgmt.example.com"
      apiconnect_fedapimgmt_runtime_id: "docker-gateway-01"
      apiconnect_fedapimgmt_instance_type: "v12"
      apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"
      apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"
      apiconnect_fedapimgmt_verify_ssl: true
      apiconnect_fedapimgmt_runtime_name: "Docker Gateway"
      apiconnect_fedapimgmt_runtime_deployment_type: "CLOUD"
      apiconnect_fedapimgmt_asset_sync_interval: 60
      apiconnect_fedapimgmt_runtime_heartbeat_interval_seconds: 60
```

**Step 3 — Build and run**

```bash
# Build the custom image
docker build -f Dockerfile.fedapimgmt -t mcpgateway-fedapimgmt:latest .

# Run with credentials passed as environment variables
docker run -d \
  --name mcpgateway \
  -p 4444:4444 \
  -e PLUGINS_ENABLED=true \
  -e PLUGINS_CONFIG_FILE=/app/plugins/config.yaml \
  -e APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER="your-client-id" \
  -e APICONNECT_FEDAPIMGMT_AUTH_SECRET="your-api-key" \
  mcpgateway-fedapimgmt:latest
```

**Step 4 — Verify**

```bash
docker logs mcpgateway 2>&1 | grep -i "fedapimgmt\|plugin\|runtime"
```

---

### Docker Compose Deployment

Use this approach when running ContextForge with Docker Compose, typically with supporting services such as a database or Redis.

**Step 1 — Create `Dockerfile.fedapimgmt`** (same as the Docker section above)

**Step 2 — Create `plugins/config.yaml`** (same as the Docker section above)

**Step 3 — Create or update `docker-compose.yml`**

```yaml
version: "3.9"

services:
  mcpgateway:
    build:
      context: .
      dockerfile: Dockerfile.fedapimgmt
    image: mcpgateway-fedapimgmt:latest
    container_name: mcpgateway
    ports:
      - "4444:4444"
    environment:
      PLUGINS_ENABLED: "true"
      PLUGINS_CONFIG_FILE: "/app/plugins/config.yaml"
      # IBM APIConnect FEDAPIMGMT credentials — use a .env file or Docker secrets for production
      APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"
      APICONNECT_FEDAPIMGMT_AUTH_SECRET: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"
    volumes:
      # Mount live config for easy updates without rebuilding
      - ./plugins/config.yaml:/app/plugins/config.yaml:ro
      # Mount TLS certificates if needed
      # - ./certs:/etc/mcpgateway/certs:ro
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4444/health"]
      interval: 30s
      timeout: 10s
      retries: 3

  # Optional: Add a database, Redis, etc.
  # postgres:
  #   image: postgres:15
  #   ...
```

**Step 4 — Set credentials in `.env`**

Create a `.env` file next to `docker-compose.yml` (never commit this file):

```bash
APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER=your-client-id-here
APICONNECT_FEDAPIMGMT_AUTH_SECRET=your-api-key-here
```

**Step 5 — Deploy**

```bash
docker compose up -d

# Stream logs
docker compose logs -f mcpgateway | grep -i fedapimgmt
```

**Step 6 — Restart after config changes**

```bash
docker compose restart mcpgateway
```

---

### Kubernetes and Helm Deployment

Use this approach when deploying MCP ContextForge to a Kubernetes cluster, typically using the official `mcp-context-forge` Helm chart.

**Step 1 — Build a custom image with the plugin pre-installed**

Create `Dockerfile.fedapimgmt`:

```dockerfile
FROM ghcr.io/ibm/mcp-context-forge:latest
RUN pip install --no-cache-dir contextforge-apiconnect-fedapimgmt
```

Build and push to your container registry:

```bash
docker build -f Dockerfile.fedapimgmt -t your-registry/mcpgateway-fedapimgmt:1.0.0 .
docker push your-registry/mcpgateway-fedapimgmt:1.0.0
```

**Step 2 — Create a Kubernetes Secret for IBM APIConnect Federated API Management credentials**

```bash
kubectl create secret generic fedapimgmt-credentials \
  --namespace mcp-stack \
  --from-literal=APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER="your-client-id" \
  --from-literal=APICONNECT_FEDAPIMGMT_AUTH_SECRET="your-api-key"
```

For TLS certificates, create a separate secret:

```bash
kubectl create secret generic fedapimgmt-tls-certs \
  --namespace mcp-stack \
  --from-file=truststore.pem=./certs/ca-bundle.pem \
  --from-file=client-cert.pem=./certs/client-cert.pem \
  --from-file=client-key.pem=./certs/client-key.pem
```

**Step 3 — Create a ConfigMap for the plugin configuration**

`fedapimgmt-plugin-configmap.yaml`:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fedapimgmt-plugin-config
  namespace: mcp-stack
data:
  config.yaml: |
    plugins:
      - name: "APIConnectFAM"
        kind: "contextforge_apiconnect_fam.apiconnect_fam.APIConnectFAMPlugin"
        description: "IBM API Connect FAM integration"
        version: "1.0.0"
        author: "Your Name"
        hooks: []
        tags: ["fedapimgmt", "sync", "monitoring"]
        mode: "permissive"
        priority: 1000
        config:
          apiconnect_fedapimgmt_enabled: true
          apiconnect_fedapimgmt_base_url: "https://fedapimgmt.prod.example.com"
          apiconnect_fedapimgmt_runtime_id: "k8s-prod-gateway-01"
          apiconnect_fedapimgmt_instance_type: "v12"
          apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"
          apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"
          apiconnect_fedapimgmt_verify_ssl: true
          apiconnect_fedapimgmt_tls_truststore_path: "/etc/mcpgateway/certs/truststore.pem"
          apiconnect_fedapimgmt_tls_truststore_type: "PEM"
          apiconnect_fedapimgmt_runtime_name: "Production K8s Gateway"
          apiconnect_fedapimgmt_runtime_deployment_type: "CLOUD"
          apiconnect_fedapimgmt_runtime_region: "us-east-1"
          apiconnect_fedapimgmt_runtime_host: "k8s-prod.example.com"
          apiconnect_fedapimgmt_runtime_tags: ["production", "kubernetes", "us-east"]
          apiconnect_fedapimgmt_asset_sync_enabled: true
          apiconnect_fedapimgmt_asset_sync_interval: 60
          apiconnect_fedapimgmt_runtime_heartbeat_interval_seconds: 60
          apiconnect_fedapimgmt_metrics_sync_enabled: true
          apiconnect_fedapimgmt_metrics_sync_interval: 300
```

Apply it:

```bash
kubectl apply -f fedapimgmt-plugin-configmap.yaml
```

**Step 4 — Create a Helm values override file**

`values-fedapimgmt.yaml`:

```yaml
image:
  repository: your-registry/mcpgateway-fedapimgmt
  tag: "1.0.0"
  pullPolicy: IfNotPresent

env:
  PLUGINS_ENABLED: "true"
  PLUGINS_CONFIG_FILE: "/app/plugins/config.yaml"

envFrom:
  - secretRef:
      name: fedapimgmt-credentials

extraVolumes:
  - name: fedapimgmt-plugin-config
    configMap:
      name: fedapimgmt-plugin-config
  - name: fedapimgmt-tls-certs
    secret:
      secretName: fedapimgmt-tls-certs

extraVolumeMounts:
  - name: fedapimgmt-plugin-config
    mountPath: /app/plugins/config.yaml
    subPath: config.yaml
    readOnly: true
  - name: fedapimgmt-tls-certs
    mountPath: /etc/mcpgateway/certs
    readOnly: true
```

**Step 5 — Install or upgrade the Helm release**

```bash
# First-time install
helm install mcp-stack oci://ghcr.io/ibm/mcp-context-forge/charts/mcp-context-forge \
  --namespace mcp-stack \
  --create-namespace \
  --values values-fedapimgmt.yaml

# Upgrade an existing release
helm upgrade mcp-stack oci://ghcr.io/ibm/mcp-context-forge/charts/mcp-context-forge \
  --namespace mcp-stack \
  --values values-fedapimgmt.yaml \
  --reuse-values
```

**Step 6 — Verify**

```bash
# Check pod status
kubectl get pods -n mcp-stack

# Tail the gateway logs
kubectl logs -n mcp-stack deployment/mcp-stack-mcpgateway -f | grep -i fedapimgmt

# Check the plugin is installed in the pod
kubectl exec -n mcp-stack deployment/mcp-stack-mcpgateway -- \
  pip show contextforge-apiconnect-fedapimgmt
```

---

## Configuration Reference

All plugin configuration lives under the `config:` key in `plugins/config.yaml`.

### Plugin Registration

The following fields are part of the cpex plugin manifest (not IBM APIConnect Federated API Management-specific):

```yaml
plugins:
  - name: "APIConnectFAM"                                                         # Display name
    kind: "contextforge_apiconnect_fam.apiconnect_fam.APIConnectFAMPlugin"         # Python class path
    description: "IBM API Connect Federated API Management integration"
    version: "1.0.0"
    author: "Your Name"
    hooks: []                 # No request hooks — plugin uses background tasks only
    tags: ["fedapimgmt", "sync"]
    mode: "permissive"        # Plugin failure does not stop the gateway
    priority: 1000
    config:
      # ... all IBM APIConnect FEDAPIMGMT-specific parameters below
```

---

### Required Parameters

| Parameter | Type | Description |
|---|---|---|
| `apiconnect_fedapimgmt_enabled` | bool | Set to `true` to activate the integration. Default: `false`. |
| `apiconnect_fedapimgmt_base_url` | string | Base URL of the IBM APIConnect Federated API Management API, without trailing slash. Example: `https://fam.example.com` |
| `apiconnect_fedapimgmt_runtime_id` | string | Unique identifier for this ContextForge instance in IBM APIConnect Federated API Management. Example: `prod-gateway-us-east-01` |
| `apiconnect_fedapimgmt_instance_type` | string | Deployment type controlling auth: `"self-hosted"` (Basic Auth), `"v12"` (Bearer token), or `"IW"` (IBM Watson Bearer token). Default: `"self-hosted"` |

---

### Authentication Parameters

| Parameter | Type | Description |
|---|---|---|
| `apiconnect_fedapimgmt_auth_identifier` | string | Role depends on `instance_type`: username (`self-hosted`), client_id (`v12`), instance_id (`IW`) |
| `apiconnect_fedapimgmt_auth_secret` | string | Role depends on `instance_type`: password (`self-hosted`), api_key (`v12`), api_key (`IW`) |
| `apiconnect_fedapimgmt_iw_token_url` | string | Token service host URL — **required when `instance_type` is `IW`**. Example: `https://account-iam.platform.saas.ibm.com` |

---

### Runtime Metadata Parameters

These parameters describe the gateway instance to IBM APIConnect Federated API Management operators. They are only used during initial registration.

| Parameter | Type | Default | Description |
|---|---|---|---|
| `apiconnect_fedapimgmt_runtime_name` | string | `"ContextForge Gateway"` | Human-readable name shown in IBM APIConnect Federated API Management |
| `apiconnect_fedapimgmt_runtime_description` | string | `"ContextForge MCP Gateway Runtime"` | Longer description |
| `apiconnect_fedapimgmt_runtime_deployment_type` | string | `"ON_PREMISE"` | Deployment type: `ON_PREMISE`, `CLOUD`, or `HYBRID` |
| `apiconnect_fedapimgmt_runtime_region` | string | `null` | Region identifier (e.g., `us-east-1`) |
| `apiconnect_fedapimgmt_runtime_location` | string | `null` | Human-readable location (e.g., `AWS US East`) |
| `apiconnect_fedapimgmt_runtime_host` | string | `null` | Host identifier (e.g., `gateway-01.prod.example.com`) |
| `apiconnect_fedapimgmt_runtime_tags` | list | `["contextforge", "mcp"]` | Tags for discovery and filtering in IBM APIConnect Federated API Management |
| `apiconnect_fedapimgmt_runtime_capacity_value` | string | `"100"` | Throughput capacity value |
| `apiconnect_fedapimgmt_runtime_capacity_unit` | string | `"per minute"` | Unit for capacity (e.g., `per minute`, `per second`) |

---

### Synchronization Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `apiconnect_fedapimgmt_asset_sync_enabled` | bool | `true` | Enable synchronization of Virtual Servers and Tools to IBM APIConnect Federated API Management |
| `apiconnect_fedapimgmt_asset_sync_interval` | int | `60` | How often (seconds) to run the asset sync activity |
| `apiconnect_fedapimgmt_runtime_heartbeat_interval_seconds` | int | `60` | How often (seconds) to send a heartbeat to IBM APIConnect Federated API Management |
| `apiconnect_fedapimgmt_metrics_sync_enabled` | bool | `false` | Enable metrics reporting to IBM APIConnect Federated API Management |
| `apiconnect_fedapimgmt_metrics_sync_interval` | int | `300` | How often (seconds) to collect and send metrics |
| `apiconnect_fedapimgmt_timeout` | int | `30` | HTTP request timeout in seconds |

---

### TLS/SSL Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `apiconnect_fedapimgmt_verify_ssl` | bool | `true` | Verify the IBM APIConnect Federated API Management server's TLS certificate. Always `true` in production. |
| `apiconnect_fedapimgmt_tls_truststore_path` | string | `null` | Path to a PEM CA bundle used to verify the IBM APIConnect Federated API Management server certificate. Uses the system CA bundle if omitted. |
| `apiconnect_fedapimgmt_tls_truststore_password` | string | `null` | Password for the truststore (not needed for PEM format). |
| `apiconnect_fedapimgmt_tls_truststore_type` | string | `"PEM"` | Format of the truststore: `PEM`, `JKS`, or `PKCS12`. |
| `apiconnect_fedapimgmt_tls_keystore_path` | string | `null` | Path to the client certificate file for Mutual TLS (mTLS). |
| `apiconnect_fedapimgmt_tls_keystore_password` | string | `null` | Password for the keystore / private key. Required if `apiconnect_fedapimgmt_tls_keystore_path` is set. |
| `apiconnect_fedapimgmt_tls_keystore_type` | string | `"PEM"` | Format of the keystore: `PEM`, `JKS`, or `PKCS12`. |
| `apiconnect_fedapimgmt_tls_key_alias` | string | `null` | Certificate alias in the keystore (JKS/PKCS12 only). |
| `apiconnect_fedapimgmt_tls_key_password` | string | `null` | Private key password if different from `apiconnect_fedapimgmt_tls_keystore_password`. |

---

## Authentication

Authentication is controlled by a single `instance_type` parameter. Set `apiconnect_fedapimgmt_auth_identifier` and `apiconnect_fedapimgmt_auth_secret` — their meaning depends on the chosen type.

### self-hosted (Basic Auth)

Use for self-hosted IBM APIConnect Federated API Management deployments. Credentials are sent as an HTTP Basic Auth header on every request.

```yaml
config:
  apiconnect_fedapimgmt_instance_type: "self-hosted"
  apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"  # username
  apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"            # password
```

**Characteristics:**
- Simple to configure — no token management
- `auth_identifier` = username, `auth_secret` = password
- Credentials transmitted on every request — always use over TLS

### v12 (Bearer Token)

Use for IBM APIConnect Federated API Management v12 SaaS instances. The plugin exchanges credentials for a short-lived Bearer token and refreshes automatically.

```yaml
config:
  apiconnect_fedapimgmt_instance_type: "v12"
  apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"  # client_id
  apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"            # api_key
```

**Characteristics:**
- `auth_identifier` = client_id, `auth_secret` = api_key
- Token fetched via `POST {base_url}/api/ingress/v1/token` with `X-ClientID` and `X-APIKEY` headers
- Token refreshed proactively 5 minutes before expiry
- Recommended for v12 SaaS production deployments

### IW (IBM Watson Platform Token)

Use for IBM Watson Platform (IW) hosted instances. The plugin obtains a Bearer token from a dedicated IAM token service URL and refreshes automatically.

```yaml
config:
  apiconnect_fedapimgmt_instance_type: "IW"
  apiconnect_fedapimgmt_auth_identifier: "${APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER}"  # instance_id
  apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"            # api_key
  apiconnect_fedapimgmt_iw_token_url: "https://account-iam.platform.saas.ibm.com"   # REQUIRED
```

**Characteristics:**
- `auth_identifier` = instance_id, `auth_secret` = api_key
- Token fetched via `POST {iw_token_url}/api/2.0/services/{instance_id}/apikeys/token` with `Mcsp-ApiKey` header
- Token expiry determined from the `expiration` field (epoch seconds) in the response
- Token refreshed proactively 5 minutes before expiry
- `apiconnect_fedapimgmt_iw_token_url` is the host only (e.g. `https://account-iam.platform.saas.ibm.com`) — the path is appended automatically

---

## TLS/SSL Configuration

### Default — System CA bundle

When `apiconnect_fedapimgmt_verify_ssl` is `true` and no truststore is specified, the plugin uses the operating system's default CA certificate bundle.

```yaml
apiconnect_fedapimgmt_verify_ssl: true
```

### Custom CA Certificate

Use this when your IBM APIConnect Federated API Management instance uses a certificate signed by an internal or private CA:

```yaml
apiconnect_fedapimgmt_verify_ssl: true
apiconnect_fedapimgmt_tls_truststore_path: "/etc/ssl/certs/corporate-ca-bundle.pem"
apiconnect_fedapimgmt_tls_truststore_type: "PEM"
```

To extract and trust the IBM APIConnect Federated API Management server certificate:

```bash
echo | openssl s_client -connect fedapimgmt.example.com:443 2>/dev/null \
  | openssl x509 > /etc/ssl/certs/fedapimgmt-cert.pem

# Append to existing bundle
cat /etc/ssl/certs/fedapimgmt-cert.pem >> /etc/ssl/certs/corporate-ca-bundle.pem
```

### Mutual TLS (mTLS)

Use when IBM APIConnect Federated API Management requires the client to present a certificate for two-way authentication:

```yaml
apiconnect_fedapimgmt_verify_ssl: true
apiconnect_fedapimgmt_tls_truststore_path: "/etc/ssl/certs/ca-bundle.pem"
apiconnect_fedapimgmt_tls_truststore_type: "PEM"
apiconnect_fedapimgmt_tls_keystore_path: "/etc/ssl/certs/contextforge-client.pem"
apiconnect_fedapimgmt_tls_keystore_password: "${TLS_KEYSTORE_PASSWORD}"
apiconnect_fedapimgmt_tls_keystore_type: "PEM"
```

> **Note:** The keystore file must contain both the client certificate and the private key in PEM format. To convert from PKCS12:
>
> ```bash
> openssl pkcs12 -in keystore.p12 -out contextforge-client.pem -nodes
> chmod 600 /etc/ssl/certs/contextforge-client.pem
> ```

> **Warning:** Never set `apiconnect_fedapimgmt_verify_ssl: false` in production. This disables all TLS verification and exposes your credentials to interception.

---

## How It Works

The plugin runs five background activities inside MCP ContextForge. Each activity runs on a configurable interval and is fault-tolerant through a circuit breaker.

### Activity Overview

| Activity | Default Interval | Description |
|---|---|---|
| **Runtime Registration** | Once at startup | Registers (or re-registers) this MCP ContextForge instance as a Runtime in IBM APIConnect Federated API Management. Detects whether recovery of missed sync data is needed. |
| **Heartbeat** | Every 60 s | Sends a lightweight ping to IBM APIConnect Federated API Management to signal that this Runtime is alive. IBM APIConnect Federated API Management marks a Runtime as inactive if heartbeats stop. |
| **Server Sync** | Every 60 s | Compares ContextForge Virtual Servers against IBM APIConnect Federated API Management MCP Servers. Creates, updates, or deletes servers in IBM APIConnect Federated API Management to match the current state. |
| **Tool Sync** | Every 60 s | Synchronizes all Tools using IBM APIConnect Federated API Management bulk operations (bulk create / update / delete). Requires at least one server to be registered first. |
| **Metrics** | Every 300 s | Collects runtime metrics (request counts, error rates, latency) and reports them to IBM APIConnect Federated API Management. Disabled by default. |

### Circuit Breaker

The circuit breaker prevents repeated failed requests from overwhelming a temporarily unavailable IBM APIConnect Federated API Management instance.

| State | Behavior |
|---|---|
| **CLOSED** | Normal operation — all requests flow through to IBM APIConnect Federated API Management. |
| **OPEN** | IBM APIConnect Federated API Management is considered unavailable — requests fail immediately without hitting IBM APIConnect Federated API Management, protecting both sides. |
| **HALF_OPEN** | After the recovery timeout, a limited number of test requests are allowed through. If they succeed, the circuit closes. |

State transitions:
```
CLOSED ──(5 failures)──► OPEN ──(60 s timeout)──► HALF_OPEN ──(success)──► CLOSED
                                                          └───(failure)───► OPEN
```

---

## Troubleshooting
This section describes common issues that you might encounter when using the ContextForge API Connect FAM plugin and provides steps to diagnose and resolve them.

### Plugin not loading

**Issue:**
- No log messages from APIConnectFAMPlugin.
- The plugin is not listed among the loaded plugins.

**Resolution:**
1. Verify plugins are enabled in `.env`:
   ```bash
   grep PLUGINS_ENABLED .env
   # Expected: PLUGINS_ENABLED=true
   ```
2. Verify that the plugin configuration file path is correct.:
   ```bash
   grep PLUGINS_CONFIG_FILE .env
   ls -la plugins/config.yaml
   ```
3. Validate the YAML syntax:
   ```bash
   python -c "import yaml; yaml.safe_load(open('plugins/config.yaml'))"
   ```
4. Confirm that the plugin is installed in the same Python environment as ContextForge:
   ```bash
   pip show contextforge-apiconnect-fedapimgmt
   pip show cpex
   ```

---

### Runtime registration fails

**Error Message:**
```
ERROR: Runtime registration failed
ERROR: IBM API Connect Federated API Management API error registering runtime
```
**Resolution**
1. Verify that  `apiconnect_fedapimgmt_base_url` is configured correctly and does not include a trailing slash:
   ```bash
   curl -v https://fedapimgmt.example.com/api/assetcatalog/v2/runtimes
   ```
2. Verify that the authentication credentials are configured correctly. For more information, see [Authentication](#authentication).
3. Test authentication manually:
   ```bash
   # v12 — get a token
   curl -X POST https://fedapimgmt.example.com/api/ingress/v1/token \
     -H "X-APIKEY: $APICONNECT_FEDAPIMGMT_AUTH_SECRET" \
     -H "X-ClientID: $APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER"

   # IW — get a token
   curl -X POST https://account-iam.platform.saas.ibm.com/api/2.0/services/$APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER/apikeys/token \
     -H "Mcsp-ApiKey: $APICONNECT_FEDAPIMGMT_AUTH_SECRET"

   # self-hosted Basic Auth
   curl -u $APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER:$APICONNECT_FEDAPIMGMT_AUTH_SECRET \
     https://fedapimgmt.example.com/api/assetcatalog/v2/runtimes
   ```

---

### Authentication fails (401 Unauthorized)

**Error Message:**
```
ERROR: IBM API Connect Federated API Management API error: status=401
ERROR: Authentication failed
```
**Resolution**

1. Verify that  `apiconnect_fedapimgmt_instance_type` matches the authentication method configured in IBM API Connect Federated API Management (`self-hosted`, `v12`, or `IW`).
2. Verify that the required environment variables are available:
   ```bash
   echo $APICONNECT_FEDAPIMGMT_AUTH_IDENTIFIER
   echo $APICONNECT_FEDAPIMGMT_AUTH_SECRET
   ```
3. Verify that the API key is valid and has not expired.
4. For Docker or Kubernetes deployments, verify that the required secrets or environment variables are mounted correctly:
   ```bash
   # Docker
   docker inspect mcpgateway | grep -A5 Env
   # Kubernetes
   kubectl exec -n mcp-stack deploy/mcp-stack-mcpgateway -- env | grep FEDAPIMGMT
   ```

---

### TLS certificate verification fails

**Error Message:**
```
ERROR: SSL: CERTIFICATE_VERIFY_FAILED
ERROR: unable to get local issuer certificate
```
**Resolution**
1. Configure a custom truststore by using `apiconnect_fedapimgmt_tls_truststore_path`.For more information, see [Configure a custom CA Certificate](#custom-ca-certificate).
2. Verify that the truststore is accessible from the ContextForge process or container:
   ```bash
   openssl x509 -in /etc/ssl/certs/ca-bundle.pem -noout -text | grep -E "Subject:|Issuer:|Not After"
   openssl s_client -connect fedapimgmt.example.com:443 -CAfile /etc/ssl/certs/ca-bundle.pem
   ```
3. Important: Disable TLS verification only for temporary development or testing purposes. Do not disable TLS verification in production environments:
   ```yaml
   apiconnect_fedapimgmt_verify_ssl: false  # Development only
   ```

---

### Mutual TLS handshake fails

**Error Message:**
```
ERROR: SSL handshake failed
ERROR: client certificate required
```
**Resolution**

1. Verify that the keystore contains both the client certificate and the corresponding private key.:
   ```bash
   head -5 /etc/ssl/certs/client-cert.pem
   # Must show: -----BEGIN CERTIFICATE-----
   grep "PRIVATE KEY" /etc/ssl/certs/client-cert.pem
   # Must find a private key section
   ```
2. Test the client certificate:
   ```bash
   curl --cert /etc/ssl/certs/client-cert.pem \
        --cacert /etc/ssl/certs/ca-bundle.pem \
        https://fedapimgmt.example.com/api/assetcatalog/v2/runtimes
   ```
3. Verify the file permissions:
   ```bash
   chmod 600 /etc/ssl/certs/client-cert.pem
   ```

---

### Virtual Servers (MCP Servers) not appearing in IBM APIConnect Federated API Management

**Issue:** Virtual Servers are available in ContextForge but are not synchronized with IBM API Connect Federated API Management.

**Resolution**

1. Verify that `apiconnect_fedapimgmt_asset_sync_enabled` is set to `true`.
2. Verify that runtime registration completed successfully :
   ```bash
   grep "Runtime registered" /var/log/contextforge/contextforge.log
   ```
3. Verify that the server synchronization activity is running. :
   ```bash
   grep -i "SyncServersActivity\|Server sync" /var/log/contextforge/contextforge.log
   ```
4. If you are using multiple Gunicorn workers, verify that synchronization is running on the primary worker:

---

### Tools not appearing in IBM APIConnect Federated API Management

**Issue:** Tools are available in ContextForge but are not synchronized with IBM API Connect Federated API Management..

**Resolution**

1. Verify that at least one Virtual Server has been synchronized before tool synchronization begins.
2. Verify that server synchronization completed successfully:
   ```bash
   grep "marked as synced\|Server.*synced" /var/log/contextforge/contextforge.log
   ```
3. Verify that the tool synchronization activity is running. :
   ```bash
   grep -i "SyncToolsActivity\|Tool sync\|Bulk.*job" /var/log/contextforge/contextforge.log
   ```

---

### Circuit breaker is open

**Error Message:**
```
ERROR: Circuit breaker is OPEN
ERROR: Circuit breaker open, cannot register runtime
```

**Resolution**

1. Verify that IBM API Connect Federated API Management is reachable.:
   ```bash
   curl -I https://fedapimgmt.example.com
   ```
2. Review the error that caused the circuit breaker to open:
   ```bash
   grep -B3 "Circuit breaker" /var/log/contextforge/contextforge.log | tail -30
   ```
3. The circuit breaker automatically attempts recovery after the configured recovery timeout. Monitor the logs for recovery messages:
   ```
   INFO: Circuit breaker attempting recovery (HALF_OPEN)
   INFO: Circuit breaker closed after successful test request
   ```
4. If necessary, increase the failure threshold or recovery timeout:
   ```yaml
   circuit_breaker_failure_threshold: 10
   circuit_breaker_recovery_timeout: 120.0
   ```

---

### Enable debug logging

To enable detailed plugin logging, configure the LOG_LEVEL environment variable:

- **Local**
```bash
# Local
export LOG_LEVEL=DEBUG
make dev
- **Docker**
# Docker
docker run -e LOG_LEVEL=DEBUG ...
- **Docker compose**
# Docker Compose
# In docker-compose.yml add: LOG_LEVEL: DEBUG
docker compose restart mcpgateway
- **Kubernetes**
# Kubernetes
kubectl set env deployment/mcp-stack-mcpgateway -n mcp-stack LOG_LEVEL=DEBUG
```

To display only plugin-related log messages:

```bash
tail -f logs/mcpgateway.log | grep -i "apiconnect\|fedapimgmt\|circuit\|heartbeat\|sync"
```

---

### Collecting diagnostic information

Before opening a support request, collect the following information:

- Plugin version
```bash
# Plugin version
pip show contextforge-apiconnect-fedapimgmt
```
- Python version and dependency versions
```bash
# Python and dependency versions
python --version
pip list | grep -E "contextforge|cpex|httpx|pydantic"
```
- Recent log files (after removing sensitive information)
```bash
# Recent logs (sanitize before sharing — remove credentials)
tail -200 /var/log/contextforge/contextforge.log > diag-logs.txt
```

- Environment configuration
```bash
# Environment check
env | grep -E "FEDAPIMGMT|PLUGINS|LOG_LEVEL" | sed 's/=.*/=REDACTED/'
```
- Steps to reproduce the issue
- IBM APIConnect Federated API Management version


---

## Security Best Practices

**Never hardcode credentials in config files**

```yaml
# Wrong
apiconnect_fedapimgmt_auth_secret: "abc123def456..."

# Correct
apiconnect_fedapimgmt_auth_secret: "${APICONNECT_FEDAPIMGMT_AUTH_SECRET}"
```

**Always enable TLS verification** — set `apiconnect_fedapimgmt_verify_ssl: true` and provide a truststore if needed. Disabling TLS verification exposes your credentials to man-in-the-middle attacks.

**Protect private key files** — restrict permissions on any TLS key material:

```bash
chmod 600 /etc/ssl/private/client-key.pem
chown contextforge:contextforge /etc/ssl/private/client-key.pem
```

**Use Kubernetes Secrets or Docker secrets** for credentials, not ConfigMaps or environment variables baked into images.

**Rotate credentials regularly** — the plugin handles token refresh transparently when you rotate the `auth_secret`.

**Monitor for authentication failures** — a sudden spike in 401 errors usually indicates an expired or revoked credential:

```bash
grep -E "(401|403|Authentication failed)" /var/log/contextforge/contextforge.log | tail -20
```

---

## License

Apache License 2.0. See [LICENSE](LICENSE) for full details.

---

## Support

- **ContextForge Repository:** [https://github.com/IBM/mcp-context-forge](https://github.com/IBM/mcp-context-forge)
- **PyPI Package:** [https://pypi.org/project/contextforge-apiconnect-fedapimgmt/](https://pypi.org/project/contextforge-apiconnect-fedapimgmt/)
