Metadata-Version: 2.4
Name: sfs-mcp-toolkit
Version: 0.1.0
Summary: Spatial Faithfulness Score (SFS) — a domain-agnostic MCP toolkit for measuring context faithfulness in RAG systems
Project-URL: Homepage, https://github.com/rganushachadika/sfs-mcp-toolkit
Project-URL: Repository, https://github.com/rganushachadika/sfs-mcp-toolkit
Project-URL: Issues, https://github.com/rganushachadika/sfs-mcp-toolkit/issues
Author-email: Ganusha Rathnasinghe <rganushachadika@gmail.com>
License: MIT License
        
        Copyright (c) 2025 Ganusha Chadika
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
License-File: LICENSE
Keywords: LLM,MCP,RAG,faithfulness,hallucination,spatial,verification
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: mcp>=1.0.0
Requires-Dist: pydantic>=2.0.0
Provides-Extra: dev
Requires-Dist: pytest; extra == 'dev'
Requires-Dist: pytest-asyncio; extra == 'dev'
Provides-Extra: server
Requires-Dist: starlette>=0.36.0; extra == 'server'
Requires-Dist: uvicorn>=0.27.0; extra == 'server'
Description-Content-Type: text/markdown

# SFS MCP Toolkit

**Spatial Faithfulness Score (SFS)** — a domain-agnostic metric and MCP server for measuring whether LLM-generated reasoning is grounded in retrieved context, not hallucinated.

## The Problem

Retrieval-Augmented Generation (RAG) systems retrieve evidence and feed it to an LLM to produce grounded answers. But how do you *measure* whether the LLM actually used that evidence faithfully, or fabricated plausible-sounding numbers?

Existing faithfulness metrics (RAGAS, DeepEval, etc.) rely on an LLM-as-judge approach — asking another LLM whether the output is faithful. This creates a circularity problem: you're using the same class of system to audit itself.

## The Solution: SFS

SFS takes a **deterministic, evidence-grounded** approach:

```
SFS = verified_claims / total_claims
```

1. **Extract** — Scan the LLM output for numerical assertions (prices, percentages, distances, counts, dosages, ratios, etc.)
2. **Verify** — For each claim, check whether the stated value falls within a configurable tolerance of any value in the evidence pool
3. **Score** — The ratio of verified claims to total claims gives a 0–1 faithfulness score

### Why This Works for Any RAG System

The key insight is that **any RAG system that produces numerical claims can be audited this way**, regardless of domain:

| Domain | Claim Types | Evidence Source |
|--------|-------------|-----------------|
| Property Valuation | Prices, gradients, distances | Comparable sales DB, spatial statistics |
| Medical RAG | Dosages, lab values, durations | Clinical databases, drug references |
| Legal RAG | Monetary amounts, durations, counts | Case law, statutory databases |
| Financial Analysis | Prices, returns, ratios | Market data, financial statements |

The SFS formula remains identical. Only the **extraction patterns** and **tolerance thresholds** change per domain — and these are fully configurable via `DomainConfig`.

### Advantages Over LLM-as-Judge

| Property | LLM-as-Judge | SFS |
|----------|-------------|-----|
| Deterministic | No — varies across runs | Yes — same input = same score |
| Auditable | Opaque reasoning | Every claim traced to evidence |
| Cost | Requires additional LLM calls | Zero LLM cost (regex + arithmetic) |
| Circular bias | Yes — LLM evaluating LLM | No — purely evidence-based |
| Reproducible | Low inter-run agreement | Perfect reproducibility |

### Validated Calibration

SFS was rigorously validated in the property valuation domain:
- **Inter-rater reliability**: Cohen's κ = 0.687 (substantial agreement) between automated SFS and expert human raters
- **Rounding tolerance**: 15% threshold correctly distinguishes acceptable rounding (68% vs 68.5%) from hallucination (50% vs 68.5%)
- **Tested across 5 prompting strategies** (zero-shot, CoT, S-CoT, few-shot, self-reflection) with consistent behaviour

## Installation

```bash
pip install sfs-mcp-toolkit
# or from source:
cd sfs-mcp-toolkit && pip install -e .
```

## MCP Server Setup

### Claude Desktop / Claude Code

Add to your MCP configuration:

```json
{
  "mcpServers": {
    "sfs": {
      "command": "sfs-mcp",
      "description": "SFS — RAG faithfulness metric"
    }
  }
}
```

### Available Tools

| Tool | Description |
|------|-------------|
| `sfs_compute` | **Primary tool** — end-to-end: extract claims + verify + score |
| `sfs_extract_claims` | Extract numerical claims from LLM output |
| `sfs_verify_claims` | Verify claims against an evidence pool |
| `sfs_verify_single` | Verify a single claim |
| `sfs_list_presets` | List available domain presets |
| `sfs_get_preset` | Get full config for a preset |

## Quick Start

### End-to-End Scoring (via MCP tool)

```
Tool: sfs_compute
text: "The median house price in this suburb is $850,000, located 12.5 km
       from the CBD. With 190 comparable sales showing a 3.2% gradient
       north, the estimated value is $875,000."
evidence_json: [
    {"key": "median_price",    "value": 842000, "unit": "AUD"},
    {"key": "cbd_distance",    "value": 12.3,   "unit": "km"},
    {"key": "comparable_count","value": 188,     "unit": "count"},
    {"key": "north_gradient",  "value": 3.5,     "unit": "%"},
    {"key": "ml_prediction",   "value": 880000,  "unit": "AUD"}
]
preset: "property_valuation"
```

**Output:**
```json
{
  "sfs_score": 1.0,
  "total_claims": 5,
  "verified_claims": 5,
  "failed_claims": 0,
  "type_breakdown": {
    "currency":   {"total": 2, "verified": 2, "failed": 0},
    "distance":   {"total": 1, "verified": 1, "failed": 0},
    "count":      {"total": 1, "verified": 1, "failed": 0},
    "percentage": {"total": 1, "verified": 1, "failed": 0}
  }
}
```

### Python API (Direct Usage)

```python
from sfs_toolkit.extractor import extract_claims
from sfs_toolkit.verifier import compute_sfs
from sfs_toolkit.models import EvidencePool, EvidenceItem
from sfs_toolkit.presets import get_preset

# Configure for your domain
config = get_preset("medical")

# Your LLM output
text = "The patient's HbA1c level of 7.2% indicates moderate control. \
        With 45 patients in the cohort showing similar patterns..."

# Your evidence pool (from RAG retrieval)
evidence = EvidencePool(items=[
    EvidenceItem(key="hba1c", value=7.1, unit="%", source="lab_results"),
    EvidenceItem(key="cohort_size", value=44, unit="count", source="study_db"),
])

# Extract and verify
claims = extract_claims(text, config=config)
report = compute_sfs(claims, evidence, config=config)

print(f"SFS: {report.sfs_score:.0%}")  # SFS: 100%
print(f"Verified: {report.verified_claims}/{report.total_claims}")
```

### Custom Domain Configuration

```python
from sfs_toolkit.models import DomainConfig, ExtractionPattern

config = DomainConfig(
    name="Climate Science RAG",
    tolerance=0.10,  # 10% tolerance
    type_tolerances={"temperature": 0.05},  # tighter for temperature
    extraction_patterns=[
        ExtractionPattern(
            claim_type="temperature",
            pattern=r'([\d.]+)\s*°[CF]',
            unit="°C",
            min_value=-100,
            max_value=200,
        ),
        ExtractionPattern(
            claim_type="concentration",
            pattern=r'([\d.]+)\s*ppm',
            unit="ppm",
            min_value=0.1,
        ),
    ],
)
```

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│                    MCP Client                           │
│  (Claude Desktop, Claude Code, any MCP-compatible app)  │
└──────────────────────┬──────────────────────────────────┘
                       │ MCP Protocol (stdio)
┌──────────────────────▼──────────────────────────────────┐
│                  SFS MCP Server                         │
│                                                         │
│  ┌─────────────┐  ┌──────────────┐  ┌───────────────┐  │
│  │  Extractor   │→│   Verifier    │→│  SFS Report    │  │
│  │ (regex-based)│  │ (tolerance-  │  │ (score +      │  │
│  │              │  │  based)      │  │  breakdown)   │  │
│  └─────────────┘  └──────────────┘  └───────────────┘  │
│         ↑                                               │
│  ┌─────────────┐                                        │
│  │   Presets    │  property_valuation | medical |        │
│  │             │  legal | financial | custom             │
│  └─────────────┘                                        │
└─────────────────────────────────────────────────────────┘
```

## Domain Presets

| Preset | Tolerance | Claim Types | Use Case |
|--------|-----------|-------------|----------|
| `property_valuation` | 15% | currency, percentage, distance, count | Real estate RAG (GeoScribe) |
| `medical` | 10% (5% for dosages) | dosage, lab_value, percentage, count, duration | Clinical decision support |
| `legal` | 10% (5% for currency) | currency, percentage, duration, count | Legal research RAG |
| `financial` | 10% (5% for ratios) | currency, percentage, ratio | Financial analysis |

## Thesis Framing

SFS was developed as part of the GeoScribe research project (FYP, IIT) for spatial property valuation. However, its design is intentionally domain-agnostic:

1. **The metric itself** (verified/total claims) is universal
2. **The verification logic** (relative error within tolerance) applies to any numerical assertion
3. **The extraction layer** is configurable via domain presets
4. **The calibration methodology** (inter-rater reliability) is reproducible in any domain

This positions SFS as a **general contribution to RAG evaluation methodology**, not just a property-valuation artifact. The GeoScribe implementation serves as a validated reference implementation, while the MCP toolkit enables adoption across domains.

## License

MIT
