Metadata-Version: 2.3
Name: sfi-toolkit
Version: 0.0.2
Summary: sfi-toolkit: Semantic Fault Injection (SFI) framework for testing AI agent resilience.
Author: Chinmay Kakatkar
License: MIT License
         
         Copyright (c) 2026 Chinmay Kakatkar
         
         Permission is hereby granted, free of charge, to any person obtaining a copy
         of this software and associated documentation files (the "Software"), to deal
         in the Software without restriction, including without limitation the rights
         to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
         copies of the Software, and to permit persons to whom the Software is
         furnished to do so, subject to the following conditions:
         
         The above copyright notice and this permission notice shall be included in all
         copies or substantial portions of the Software.
         
         THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
         IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
         FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
         AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
         LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
         OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
         SOFTWARE.
Requires-Dist: jsonschema>=4.0
Requires-Dist: scipy>=1.10
Requires-Python: >=3.12
Project-URL: Homepage, https://github.com/ckstash/sfi-toolkit
Description-Content-Type: text/markdown

# sfi-toolkit

**sfi-toolkit** is a Python library for Semantic Fault Injection (SFI) — a testing methodology for LLM-based agent resilience.

---

## Features

- **Rule-based semantic fault generator** — deterministically perturbs correct tool responses into wrong-period, wrong-entity, or numeric-noise faults. All faults are schema-valid and statistically plausible.
- **Three hardening strategies** — H1 (cross-source validation), H2 (business-rule plausibility), H3 (trajectory consistency) — independently and composably reduce fault propagation.
- **Propagation experiment runner** — simulates the agent as a Bernoulli chain and measures fault propagation rates across fault classes, injection steps, and hardening configurations.
- **Provider-agnostic LLM fault generator** — accepts any `Callable[[str], str]` as the LLM backend; no SDK dependency.

---

## Installation

```bash
pip install sfi-toolkit
```

Built for Python 3.12 or above.

---

## Quick Start

### 1. Generate a wrong-period fault

```python
from sfi_toolkit import FaultClass, RuleBasedFaultGenerator

schema = {
    "type": "object",
    "required": ["company_code", "fiscal_period", "accounts"],
    "properties": {
        "company_code": {"type": "string"},
        "fiscal_period": {"type": "string"},
        "accounts": {"type": "array"},
    },
}

correct_response = {
    "company_code": "1000",
    "fiscal_period": "2025-Q3",
    "accounts": [{"account_id": "110000", "balance": 1842350.00}],
}

gen = RuleBasedFaultGenerator(
    schema=schema,
    entity_catalog=["1000", "2000", "3000", "4000", "5000"],
)
fault = gen.generate(correct_response, FaultClass.WRONG_PERIOD)
print(fault.perturbed_response["fiscal_period"])  # "2025-Q2"
print(fault.schema_valid)                          # True
```

### 2. Check a response with H3 trajectory consistency

```python
from sfi_toolkit import H3Consistency

h3 = H3Consistency()
ctx = {"company_code": "1000", "fiscal_period": "2025-Q3"}

# Record what we expect from the first step
h3.observe(correct_response, ctx)

# A later step returns data for the wrong period
wrong = dict(correct_response)
wrong["fiscal_period"] = "2025-Q2"
result = h3.check(wrong, ctx)
print(result.detected)  # True
print(result.reason)    # "fiscal_period mismatch: ..."
```

### 3. Run a propagation experiment

```python
from sfi_toolkit import PropagationExperiment

exp = PropagationExperiment()
faults = gen.generate_batch(correct_response, FaultClass.WRONG_PERIOD, n=50)

result_h0 = exp.run(faults, injection_step=1, hardening=set())
result_all = exp.run(faults, injection_step=1, hardening={"H1", "H2", "H3"})

print(f"H0  propagation rate: {result_h0.propagation_rate:.2f}")   # ~0.68
print(f"All propagation rate: {result_all.propagation_rate:.2f}")  # ≤ 0.05
```

---

## Fault Taxonomy

| | **Innocent** | **Adversarial** |
|---|---|---|
| **User-side** | Usability testing | Red teaming |
| **Environment-side** | **SFI (this library)** | Indirect prompt injection |

SFI targets the environment-side innocent cell — the gap not addressed by any prior methodology.

---

## API Reference

See [API.md](API.md).

---

## License

MIT License. See [LICENSE](LICENSE).
