Metadata-Version: 2.4
Name: uagf-xai
Version: 1.0.0
Summary: Constraint-based audit evidence planning and report generation for traditional ML and LLM/agentic systems.
License-Expression: MIT
Keywords: xai,ai-governance,audit-evidence,eu-ai-act,fairness,drift,uncertainty,llm-evaluation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Requires-Python: <3.13,>=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy==2.3.5
Requires-Dist: pandas==2.3.3
Requires-Dist: scipy==1.15.3
Requires-Dist: scikit-learn==1.8.0
Requires-Dist: joblib==1.5.3
Requires-Dist: Jinja2==3.1.6
Requires-Dist: matplotlib==3.10.8
Requires-Dist: shap==0.49.1
Requires-Dist: lime==0.2.0.1
Requires-Dist: dice-ml==0.12
Requires-Dist: fairlearn==0.13.0
Requires-Dist: MAPIE==1.3.0
Requires-Dist: evidently==0.7.21
Provides-Extra: llm
Requires-Dist: torch==2.11.0; extra == "llm"
Requires-Dist: transformers==5.6.0; extra == "llm"
Requires-Dist: safetensors==0.7.0; extra == "llm"
Provides-Extra: pdf
Requires-Dist: playwright==1.61.0; extra == "pdf"
Provides-Extra: model-formats
Requires-Dist: xgboost==3.2.0; extra == "model-formats"
Requires-Dist: lightgbm==4.6.0; extra == "model-formats"
Provides-Extra: dev
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: build; extra == "dev"
Requires-Dist: twine; extra == "dev"
Dynamic: license-file

# UAGF-XAI

UAGF-XAI is an MSc thesis project and the S6 audit-evidence component of the
**Unified AI Governance Framework (UAGF)**. It turns normalized governance and
technical-audit context into an auditable evidence plan, executes compatible
explainability and monitoring methods, and generates structured HTML and PDF
audit reports.

## UAGF Research Platform

UAGF is a research programme for connecting organizational AI governance,
technical audit preparation, and execution-level evidence generation. Its three
thesis components have deliberately separate responsibilities:

| Component | Project role | Main responsibility | Downstream output |
| --- | --- | --- | --- |
| S4 — UAGF-GMM/CGSA | Governance maturity | Assess organizational controls and governance readiness | Governance score, verdict, and domain scores |
| S5 — UAGF-TAM/AAA | Technical audit methodology | Establish the audit scope, system/risk context, findings, and technical resource references | Normalized audit context and resource contract |
| S6 — UAGF-XAI/CBEP | Audit evidence toolkit | Plan and execute compatible evidence methods and report their results | Structured evidence plus HTML/PDF reports |

Together, the components support a flow from governance assessment to audit
context and finally to technical evidence. They are complementary rather than
interchangeable: S4 assesses governance, S5 defines the audit engagement, and
S6 generates evidence. UAGF-XAI does not replace the upstream assessments or
make a final legal-compliance decision.

The professor-provided UAGF program map is intentionally not embedded in the
PyPI long description or distributed as package data.

The map is theoretical programme material. The implementation described below
preserves its high-level division of responsibility while documenting the
practical contract and validation adjustments made during development.

### Upstream S4: UAGF-GMM/CGSA

S4 evaluates governance maturity across the UAGF control domains. The S6
adapter consumes a compact `GovernanceContext` containing:

- the composite governance maturity score;
- the governance verdict;
- the per-domain maturity scores, including transparency and monitoring.

These values are planning signals. For example, CBEP can broaden the evidence
sweep when overall maturity is low, promote explainability when transparency
is weak, or promote drift evidence when monitoring maturity is weak. S6 does
not recalculate the S4 assessment.

### Upstream S5: UAGF-TAM/AAA

S5 defines the formal technical-audit context. Its normalized `AuditContext`
provides:

- system type, modality, task type, application domain, and risk tier;
- applicable EU AI Act articles, audit findings, and the upstream CSP/status
  signal;
- model artifact URI, artifact kind, format, framework, type, and optional
  directory entrypoint;
- traditional training/reference and evaluation datasets, or an LLM golden
  set and optional prompt/RAG/guardrail resources;
- target, positive-label, sensitive-feature, and counterfactual-policy
  metadata where applicable.

S5 supplies five of the six final validation cases. The sixth case is an
explicitly S6-owned local LLM execution-validation configuration and is not
represented as an S5 delivery.


## S6 Thesis Project: UAGF-XAI

UAGF-XAI is a Python audit-evidence framework. Its main contribution is the
orchestration of established evidence tools through a unified, context-aware,
reproducible pipeline. It does not claim that the underlying SHAP, LIME, DiCE,
Fairlearn, MAPIE, or Evidently algorithms were newly invented in this thesis.

The S6 workflow performs six main functions:

1. normalize S4 and S5 inputs through dedicated adapters;
2. resolve the exact model and data resources described by the audit contract;
3. use CBEP to construct a minimum-sufficient operational evidence plan;
4. execute only methods compatible with the system, task, modality, and
   available resources;
5. normalize heterogeneous outputs into one `EvidenceResult` schema;
6. generate reproducible, case-specific HTML and PDF audit reports.

The final implemented scope is **six traditional ML evidence methods plus four
professor-defined LLM/agentic methods**.

## Implemented Evidence Pathways

### Traditional ML: four evidence layers

| Layer | Method | Evidence produced | Main applicability boundary |
| --- | --- | --- | --- |
| Layer 1 — Explainability | SHAP | Global feature attribution or text token/n-gram attribution | Fitted estimator and supported model input |
| Layer 1 — Explainability | LIME | One-sample local surrogate explanation and fidelity | Current tabular/text classification and tabular regression paths |
| Layer 1 — Explainability | DiCE | Structured counterfactual explanations | Tabular classification with a probability interface |
| Layer 2 — Fairness | Fairlearn | Demographic parity, equalized odds, and group selection rates | Classification with S5-declared sensitive features |
| Layer 3 — Uncertainty | MAPIE | Conformal prediction sets or numeric intervals | Classification, regression, and forecasting with a fitted estimator |
| Layer 4 — Drift | Evidently plus feature tests | Canonical model-input distribution drift | Reference and current data with authoritative feature scope |

### LLM / Agentic pathway

The LLM pathway uses the professor-defined method names and ordering:

| ID | Method | Evidence role |
| --- | --- | --- |
| LLM-E1 | Grounding Score | Semantic alignment between generated continuation and supplied context/reference text |
| LLM-E2 | Self-Consistency Score | Semantic consistency across repeated seeded generations |
| LLM-E3 | Semantic Drift Index | Controlled semantic change between baseline and current prompt sets |
| LLM-E4 | Differential Prompt Fairness | Output sensitivity across explicitly matched demographic prompt pairs |

Aequitas, MC Dropout, automatic protected-attribute inference, external model
downloads, and fabricated LLM predictions are not part of the final scope.

## Evidence Tools: Purpose and Audit Role

The evidence methods answer different audit questions. They are intentionally
combined because no single method can explain model behaviour, assess group
outcomes, quantify predictive uncertainty, and monitor changing inputs at the
same time. Each result is technical evidence for an auditor to interpret; none
of the tools independently establishes legal compliance.

### SHAP — feature attribution

**Question answered:** Which model inputs contributed most strongly to the
model's predictions?

SHAP assigns contribution values to model features. UAGF-XAI aggregates the
absolute contribution magnitudes to produce a global ranking and saves a SHAP
visualization. For fitted traditional NLP pipelines, the same pathway reports
the original TF-IDF vocabulary tokens and n-grams instead of replacing them
with generic feature identifiers.

**Audit value:** SHAP supports transparency and model-behaviour inspection by
showing which variables dominate the fitted model's outputs. It contributes
primarily to Article 13 evidence.

**Main outputs:** ranked features or tokens/n-grams, attribution magnitudes,
explainer metadata, and a summary plot.

**Interpretation boundary:** attribution describes model behaviour, not causal
influence or the correctness of the learned relationship.

### LIME — local surrogate explanation

**Question answered:** Why did the model produce this prediction for one
particular evaluation instance?

LIME perturbs inputs around a selected sample and fits a simpler local
surrogate to approximate the original model near that point. UAGF-XAI records
the local feature or token contributions together with the surrogate fidelity
score.

**Audit value:** LIME complements SHAP with an instance-level explanation that
can help an auditor inspect an individual decision or text prediction. It also
supports Article 13 transparency evidence.

**Main outputs:** sample index, local feature/token contributions, predicted
class or value, local surrogate prediction, fidelity, and an HTML explanation
artifact.

**Interpretation boundary:** LIME explains a local neighborhood, not the whole
model. A low fidelity score means the surrogate is a weak approximation and
must be interpreted cautiously.

### DiCE — counterfactual explanation

**Question answered:** Which allowed feature changes could lead the model to a
different prediction?

DiCE searches for alternative input instances that change the model outcome.
UAGF-XAI stores every generated counterfactual in structured JSON and reports
the original prediction, counterfactual prediction, changed features, and the
validated actionable/immutable/sensitive-feature policy.

**Audit value:** counterfactuals provide a human-readable what-if view of a
classification decision and complement attribution methods in Layer 1
Explainability. They contribute to Article 13 evidence.

**Main outputs:** one or more counterfactual instances, prediction changes,
changed-feature counts and values, feature-policy metadata, and a JSON
artifact.

**Interpretation boundary:** generated changes are mathematical explanations,
not recommendations. They may be infeasible, undesirable, or non-actionable
and require domain-expert review. The current implementation supports tabular
classification, not text, forecasting, or anomaly detection.

### Fairlearn — group fairness metrics

**Question answered:** Do observed prediction outcomes differ across the
sensitive groups explicitly identified in the S5 audit context?

Fairlearn compares predictions by sensitive-feature group. UAGF-XAI calculates
demographic parity difference, equalized odds difference, and per-group
selection rates using the declared positive label.

**Audit value:** the metrics make statistical outcome disparities visible and
provide evidence relevant to Article 10 data-governance and fairness review.

**Main outputs:** evaluated sensitive columns, demographic parity difference,
equalized odds difference, and group selection rates.

**Interpretation boundary:** UAGF-XAI does not infer protected attributes.
Missing S5 sensitive-feature metadata produces an evidence gap. Statistical
disparity does not by itself prove unlawful discrimination or explain its
cause.

### MAPIE — conformal uncertainty estimation

**Question answered:** How much uncertainty surrounds the fitted model's
predictions, and does the observed conformal coverage match the requested
confidence level?

MAPIE adds conformal uncertainty evidence without retraining the audited
estimator. UAGF-XAI uses the original fitted S5 estimator in prefit mode and
divides evaluation data into disjoint conformal-calibration and measurement
subsets. Classification produces prediction sets; regression and forecasting
produce numeric prediction intervals.

**Audit value:** MAPIE makes uncertainty and coverage measurable rather than
relying only on point predictions. Its evidence supports Articles 9, 14, and
15 concerning risk, oversight, accuracy, and robustness.

**Main outputs:** confidence level, empirical coverage, coverage gap,
prediction-set size or interval width, estimator mode, and calibration policy.

**Interpretation boundary:** reported coverage depends on the available
calibration and measurement data. Forecasting intervals are marginal and do
not explicitly model temporal dependence.

### Evidently and feature drift tests — distribution monitoring

**Question answered:** Have the authoritative model-input distributions changed
between the reference/training data and the current evaluation data?

Evidently provides the canonical per-feature drift assessment when its result
can be extracted consistently. UAGF-XAI also records numeric Kolmogorov-Smirnov
and categorical Jensen-Shannon tests as supplementary or fallback evidence.
For traditional NLP, text drift is evaluated through derived document and
fitted-token statistical characteristics of the authoritative text column.

**Audit value:** drift evidence helps identify changing operating conditions
that may require investigation or post-market monitoring. It contributes to
Articles 15 and 61.

**Main outputs:** analyzed feature scope, per-feature test results, drifted
features, drift share, dataset-level drift decision, thresholds, and a
structured JSON artifact.

**Interpretation boundary:** input-distribution drift does not prove concept
drift, reduced model performance, or a compliance failure. Contextual columns
outside the fitted model input do not override the canonical model-input drift
conclusion.

### LLM-E1 — Grounding Score

**Question answered:** How closely does a generated continuation align
semantically with the supplied context or reference answer?

LLM-E1 generates a continuation for each golden-set record, embeds the
continuation and reference/context text, and calculates cosine similarity.

**Audit value:** the score provides execution-level grounding evidence for LLM
transparency and Article 13 review.

**Main outputs:** per-record grounding scores, mean, median, minimum and maximum
alignment, generation metadata, and prompt-echo diagnostics.

**Interpretation boundary:** semantic similarity is not a calibrated
probability and does not prove factual accuracy, legal correctness, or complete
support for every generated claim.

### LLM-E2 — Self-Consistency Score

**Question answered:** Does the LLM produce semantically consistent outputs when
the same prompt is sampled repeatedly?

LLM-E2 generates five seeded stochastic continuations for each configured
prompt and compares all pairwise embedding similarities. The local validation
case uses three prompts and five generations per prompt.

**Audit value:** repeated-generation consistency provides reliability and
uncertainty evidence relevant to Articles 9, 14, and 15.

**Main outputs:** prompt count, generations per prompt, seeds, pairwise semantic
similarities, per-prompt consistency, and aggregate consistency.

**Interpretation boundary:** stable outputs can still be wrong, and variable
outputs can sometimes reflect legitimate ambiguity. The score must be read
together with grounding and task context.

### LLM-E3 — Semantic Drift Index

**Question answered:** How different are generated outputs across an explicitly
paired baseline and current prompt set?

LLM-E3 generates deterministic continuations for matched baseline/current
prompts, embeds the outputs, and calculates pairwise and centroid semantic
distance. The reported Semantic Drift Index is the centroid distance.

**Audit value:** it validates the semantic monitoring pathway associated with
Articles 15 and 61.

**Main outputs:** paired generation records, pairwise semantic distances,
centroid similarity, centroid distance, embedding-model metadata, and the
Semantic Drift Index.

**Interpretation boundary:** the current S6 local case is controlled synthetic
validation, not observed production drift, and no legal threshold is inferred
from the score.

### LLM-E4 — Differential Prompt Fairness

**Question answered:** How much does the generated output change when an
explicitly matched prompt varies only a controlled demographic attribute?

LLM-E4 runs paired prompts, embeds both continuations, and reports semantic
output difference together with prompt-echo diagnostics. The fixture validates
that each pair differs only in the declared demographic substitution.

**Audit value:** differential prompt testing exposes output sensitivity relevant
to Article 10 fairness review when standard classification group metrics are
not appropriate for generative systems.

**Main outputs:** controlled attributes, matched pair records, per-pair output
similarity/difference, aggregate output difference, generation configuration,
and evidence-quality flags.

**Interpretation boundary:** output difference does not independently establish
fairness or unlawful discrimination. The current local DistilGPT2 result is
explicitly marked `limited_by_degenerate_generation` because all four pairs
exhibited prompt-echo behavior.

### How the evidence fits together

For traditional ML, the four layers answer complementary questions:

```text
Explainability -> What drove the prediction?
Fairness       -> Do outcomes differ across declared groups?
Uncertainty    -> How reliable or well-covered are predictions?
Drift          -> Has the model-input environment changed?
```

For LLM/agentic systems, LLM-E1 through LLM-E4 provide corresponding evidence
for grounding, reliability, semantic monitoring, and differential prompt
behavior. CBEP decides which applicable methods are required; the report keeps
their outputs, limitations, and execution status separate rather than merging
them into one unsupported compliance score.

## Architecture and End-to-End Workflow

The same workflow is used in the thesis notes and UAGF platform documentation.
It makes the two independent S6 inputs explicit: CBEP receives normalized
contexts, while ResourceLoader resolves execution resources. The method plan
and `ResourceBundle` meet only at the Executor boundary.

```text
┌────────────────────────┐                   ┌────────────────────────────────┐
│ Input 1:               |                   | Input 2:                       |
|   S4 Governance JSON   │                   │   S5 Audit JSON or             │
└───────────┬────────────┘                   │   S6 Local Validation Config   │
            ▼                                └───────────────┬────────────────┘
┌────────────────────────┐                                   ▼
│ Governance Adapter     │                ┌──────────────────────────────────────┐
└───────────┬────────────┘                │ Audit Adapter / Local Config Adapter │
            ▼                             └──────────────────┬───────────────────┘
┌────────────────────────┐                                   ▼
│ GovernanceContext      │                    ┌─────────────────────────────┐
└───────────┬────────────┘                    │        AuditContext         │
            │                                 └─────┬─────────────────┬─────┘
            │ planning context                      │ for planning    │ for resources loading
            └───────────────────┐        ┌──────────┘                 ▼
                                ▼        ▼                  ┌────────────────────┐
                         ┌────────────────────┐             │ ResourceLoader     │
                         │ CBEP Planner       │             └─────────┬──────────┘
                         └─────────┬──────────┘                       │
                                   │                      ┌───────────┼────────────┐
                                   │                      ▼                        ▼
                                   │                    Model Loader          Dataset Loader
                                   │                      └───────────┬────────────┘
                                   │                                  ▼
                                   ▼                        ┌────────────────────┐
                         ┌────────────────────┐             │ ResourceBundle     │
                         │ Evidence Plan      │             └─────────┬──────────┘
                         │ + Planning Trace   │                       │
                         └─────────┬──────────┘                       │
                                   └──────────────┬───────────────────┘
                                                  ▼
                                        ┌────────────────────┐
                                        │ Evidence Executor  │
                                        └─────────┬──────────┘
                                                  ▼
                                        ┌────────────────────┐
                                        │ Select Pathway     │
                                        └─────────┬──────────┘
                             ┌────────────────────┴─────────────────────┐
                             ▼                                          ▼
             ┌──────────────────────────────────┐      ┌──────────────────────────────────────┐
             │ If Traditional ML system         │      │ Else LLM / Agentic system            │
             ├──────────────────────────────────|      ├──────────────────────────────────────|
             │ Layer 1: Explainability (SHAP,   │      | LLM-E1: Grounding Score              |
             │          LIME, DiCE)             │      | LLM-E2: Self-Consistency Score       |
             │ Layer 2: Fairness (Fairlearn)    │      | LLM-E3: Semantic Drift Index         |
             │ Layer 3: Uncertainty (MAPIE)     │      | LLM-E4: Differential Prompt Fairness |
             │ Layer 4: Drift (Evidently)       │      |                                      |
             └─────────────────┬────────────────┘      └───────────┬──────────────────────────┘
                               └────────────────┬──────────────────┘
                                                ▼
                                      ┌────────────────────┐
                                      │ Evidence Normalizer│
                                      └─────────┬──────────┘
                                                ▼
                                      ┌────────────────────┐
                                      │ EvidenceResult     │
                                      └─────────┬──────────┘
                                                ▼
                                      ┌────────────────────┐
                                      │ HTML + PDF Report  │
                                      └────────────────────┘
```

`AuditContext` intentionally has two read-only consumers:

- **ResourceLoader** reads the resource-contract fields needed to locate and
  load the model, datasets, golden set, and optional LLM resources.
- **CBEP** reads the planning fields needed to select evidence, such as risk
  tier, applicable articles, system type, task type, modality, and the upstream
  CSP/status signal.

The responsibilities do not overlap. CBEP never loads artifacts, ResourceLoader
never selects evidence methods, `GovernanceContext` is used only by CBEP, and
`ResourceBundle` is passed to the Executor rather than back into the planner.

### Architectural boundaries

- Only adapters parse S4/S5 JSON.
- Only resource loaders resolve files, folders, datasets, and raw URIs.
- CBEP consumes `AuditContext` and `GovernanceContext`, not `ResourceBundle`.
- Evidence layers receive resolved objects and do not open S4/S5 resources.
- The exact fitted S5 artifact remains the source of predictions.
- Saved encoders and vectorizers are used read-only; they are not refitted.
- Traditional ML and LLM resource contracts remain separate.
- S6 local LLM configuration remains separate from the formal S5 contract.
- Evidence status is explicit: `completed`, `skipped`, `failed`, or
  `not_applicable`.

## Constraint-Based Evidence Planner

CBEP is the core thesis contribution. It is a deterministic,
constraint-informed planner inspired by constraint satisfaction theory. It
does not claim to use a generic CSP solver. The S5 `csp_satisfied` field is an
upstream audit-status signal that can broaden the sweep; it is not an S6 solver
result.

```text
                 ┌──────────────────────────────────┐
                 │ AuditContext + GovernanceContext │
                 └────────────────┬─────────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Select System Pathway            │
                 └────────────────┬─────────────────┘
                    ┌─────────────┴─────────────┐
                    ▼                           ▼
       ┌────────────────────────┐  ┌────────────────────────┐
       │ Traditional Catalogue  │  │ LLM-E1 to LLM-E4       │
       └────────────┬───────────┘  └────────────┬───────────┘
                    └─────────────┬─────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Build Risk-Tier Base Plan        │
                 └────────────────┬─────────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Add EU AI Act Requirements       │
                 └────────────────┬─────────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ S4 Governance Priority?          │
                 └────────────────┬─────────────────┘
                          ┌───────┴───────┐
                      Yes ▼               ▼ No
       ┌────────────────────────┐         │
       │ Promote Methods or     │         │
       │ Broaden Evidence Sweep │         │
       └────────────┬───────────┘         │
                    └─────────────┬───────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ S5 csp_satisfied?                │
                 └────────────────┬─────────────────┘
                          ┌───────┴───────┐
                    False ▼               ▼ True
       ┌────────────────────────┐         │
       │ Use Full Pathway       │         │
       │ Catalogue              │         │
       └────────────┬───────────┘         │
                    └─────────────┬───────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Merged Candidate Method Set      │
                 └────────────────┬─────────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Task + Modality Compatibility    │
                 └────────────────┬─────────────────┘
                    ┌─────────────┴─────────────┐
                    ▼                           ▼
       ┌────────────────────────┐  ┌────────────────────────┐
       │ Compatible Methods     │  │ Incompatible Methods   │
       └────────────┬───────────┘  │ + Exclusion Reasons    │
                    ▼              └────────────┬───────────┘
       ┌────────────────────────┐               │
       │ Stable Method Ordering │               │
       └────────────┬───────────┘               │
                    ▼                           │
       ┌────────────────────────┐               │
       │ Minimum-Sufficient     │               │
       │ Operational Plan       │               │
       └────────────┬───────────┘               │
                    └─────────────┬─────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Final Plan + Auditable Trace     │
                 └────────────────┬─────────────────┘
                                  ▼
                 ┌──────────────────────────────────┐
                 │ Evidence Executor                │
                 └──────────────────────────────────┘
```

### Planning inputs and rules

CBEP combines:

- system pathway: traditional ML or LLM/agentic;
- risk-tier base plan: minimal, limited, or high;
- EU AI Act article mappings for Articles 9, 10, 13, 14, 15, and 61;
- the S4 overall governance score and transparency/monitoring domains;
- the upstream S5 CSP/status signal;
- task and modality compatibility;
- stable catalogue ordering for execution and reporting.

The planner returns the final method tokens, display names, per-method
decisions, incompatibility reasons, and the complete planning trace.

For this project, **minimum sufficient** means the smallest compatible subset
of the implemented catalogue required by the merged rules. It is an operational
definition relative to the catalogue and rule base, not a universal proof of
legally complete or globally minimal evidence.

Method identity, compatibility, layer, evidence type, article mapping,
requirements, and report labels are centralized in
`schema/method_catalog.py`.

## Resource Contracts and Model Integrity

### Traditional ML contract

Traditional cases provide:

- a model artifact URI and explicit `single_file` or `directory` kind;
- model format, framework, type, and optional directory entrypoint;
- training/reference and evaluation datasets;
- target and optional feature, sensitive-feature, actionable-feature, and
  immutable-feature metadata.

Supported artifact structures include:

- directly serialized `.joblib`, `.pkl`, and `.pickle` estimators;
- sklearn dictionary bundles with a fitted model, saved encoders, and
  authoritative feature columns;
- model directories with explicit serialized entrypoints;
- fitted sklearn text pipelines with their original TF-IDF vectorizer and
  classifier.

### LLM / Agentic contract

LLM cases use a model directory and golden set, with optional system prompt,
RAG manifest, and guardrail configuration. They do not require traditional
training/evaluation CSV files.

A HuggingFace directory without model weights becomes a structured
metadata-only artifact. The golden set and optional resources are still
validated, while inference-level methods return explicit skipped evidence.
UAGF-XAI does not download a substitute model or fabricate predictions.

### Read-only model use

The framework audits the exact supplied artifact:

- no audited model is retrained;
- no saved encoder or TF-IDF vectorizer is fitted again;
- no classifier is reconstructed from copied coefficients;
- no manual sigmoid or softmax replaces the original `predict_proba`;
- MAPIE uses the fitted estimator in prefit mode;
- raw audit-readable data, model features, encoded/vectorized input, sensitive
  data, and target labels remain separate views.

## Five Validation Domains and Six Final Cases

The professor-defined validation design contains five domain slots. The final
repository contains six cases because the LLM domain is validated twice: once
with a formal metadata-only S5 contract and once with an executable S6 local
surrogate.

| Professor-defined domain | Final case mapping | Source | Risk | System / modality / task |
| --- | --- | --- | --- | --- |
| Finance | FinClear GmbH | S5 | High | Traditional ML / tabular / binary classification |
| Retail forecasting | RetailIQ AG | S5 | Minimal | Traditional ML / time series / forecasting |
| Energy / critical-infrastructure monitoring | HarbourLogistik GmbH | S5 | High | Traditional ML / tabular / anomaly detection |
| General NLP | TalentSift GmbH | S5 | High | Traditional ML / text / binary classification |
| LLM / Agentic | LegalMind AI Ltd and S6 local DistilGPT2 | S5 plus S6 local | High | Agentic/LLM / text / generation |

The HarbourLogistik case is the practical implementation used for the original
energy/operational-monitoring validation slot. It audits critical-infrastructure
sensor anomaly detection rather than the early Exposé example of tabular energy
regression. This implementation adjustment should be stated explicitly in the
thesis.

### Case 1 — FinClear GmbH: Finance

- **Contract:** formal S5 traditional ML, single-file joblib artifact.
- **Model:** sklearn `GradientBoostingClassifier` bundle with saved categorical
  encoders and 20 authoritative model features.
- **Data:** 700 reference/training rows and 300 evaluation rows; 21 columns
  including the `credit_risk` target.
- **Sensitive features:** `personal_status` and `foreign_worker`.
- **CBEP plan:** SHAP, LIME, DiCE, Fairlearn, MAPIE, and Drift.
- **Outcome:** all six methods completed; HTML/PDF and method artifacts were
  generated.

### Case 2 — RetailIQ AG: Retail forecasting

- **Contract:** formal S5 traditional ML, model directory with the explicit
  `demandpulse_v3.0.joblib` entrypoint.
- **Model:** Chronos/sklearn forecasting wrapper.
- **Data:** 1,540 reference/training rows and 385 evaluation rows; 12 columns
  including the `sales` target.
- **Sensitive features:** none declared because group-classification fairness
  is not applicable to this forecasting task.
- **CBEP plan after compatibility filtering:** SHAP, MAPIE, and Drift.
- **Outcome:** all three selected methods completed; LIME, DiCE, and Fairlearn
  were recorded as not applicable.

### Case 3 — HarbourLogistik GmbH: Energy/critical infrastructure

- **Contract:** formal S5 traditional ML, single-file pickle bundle.
- **Model:** fitted `IsolationForest` anomaly detector with 13 authoritative
  model features.
- **Data:** 5,000 reference/training rows and 500 evaluation rows; 16 audit
  dataset columns, with non-model contextual/target columns kept outside the
  canonical model-input drift scope.
- **Sensitive features:** none declared.
- **CBEP plan after compatibility filtering:** SHAP and Drift.
- **Outcome:** both selected methods completed; probability- and
  classification-dependent methods were recorded as not applicable.

### Case 4 — TalentSift GmbH: General NLP

- **Contract:** formal S5 traditional ML, single-file joblib artifact.
- **Model:** fitted TF-IDF plus `LogisticRegression` sklearn pipeline; the
  authoritative model input is `cv_text`.
- **Data:** 240 reference/training rows and 80 evaluation rows; seven audit
  columns including the target and protected/context columns.
- **Sensitive features:** `sex`, `age_group`, and `nationality`.
- **CBEP plan after compatibility filtering:** SHAP, LIME, Fairlearn, MAPIE,
  and Drift. DiCE is excluded because the current counterfactual implementation
  is tabular.
- **Outcome:** all five selected methods completed. SHAP reports fitted
  vocabulary tokens and n-grams, while text drift evaluates derived
  characteristics of the authoritative `cv_text` input.

### Case 5 — LegalMind AI Ltd: formal LLM/Agentic contract

- **Contract:** formal S5 LLM/agentic golden-set contract; no traditional CSV
  datasets are required.
- **Model artifact:** metadata-only Mistral/LoRA/RAG directory containing
  configuration files but no loadable weights.
- **Data:** 60 golden evaluation records, plus optional system-prompt, RAG, and
  guardrail resources.
- **CBEP plan:** LLM-E1, LLM-E2, LLM-E3, and LLM-E4.
- **Outcome:** the contract and resources were validated, and all four methods
  returned structured skipped evidence because inference was unavailable. No
  model was downloaded and no predictions were faked.

### Case 6 — S6 local DistilGPT2: LLM execution validation

- **Contract:** S6-owned local mode, separate from the formal S5 DTO.
- **Models:** locally stored DistilGPT2 generator and locally stored
  `all-MiniLM-L6-v2` evaluation embedding model.
- **Evaluation fixtures:** three grounding records, three self-consistency
  prompts, two controlled semantic-drift pairs, and four matched fairness
  prompt pairs.
- **CBEP plan:** LLM-E1, LLM-E2, LLM-E3, and LLM-E4.
- **Outcome:** all four methods completed with continuation-only generation.
  E4 is explicitly marked as limited by degenerate prompt-echo generation.
- **Interpretation:** this validates LLM evidence-path mechanics; the scores are
  not an audit of the LegalMind model.

## Unified Evidence and Reports

All runners cross the reporting boundary through the same schema:

```text
evidence_id, layer, method, status, article_mapping,
summary, key_findings, metrics, artifacts, limitations, raw_output
```

HTML and PDF reports include:

- Executive Summary;
- Audit Scope and Input Context;
- Resource Loading Summary;
- Runtime and Reproducibility Information;
- CBEP Evidence Planning Summary;
- Evidence Coverage Matrix;
- Evidence Findings by Layer;
- Raw Evidence Appendix.

Input provenance uses project-relative paths and SHA-256 hashes. Traditional
reports use the stable order SHAP, LIME, DiCE, Fairlearn, MAPIE, and Drift. LLM
reports use LLM-E1 through LLM-E4.

Final reports are written under `outputs/report/`; supporting method artifacts
are written under directories such as `outputs/shap/`, `outputs/lime/`,
`outputs/dice/`, and `outputs/drift/`.

## Repository Structure

```text
adapters/      S4 and S5 normalization
api/           orchestration boundary
resources/     artifact, model, dataset, LLM, and local-config loading
planner/       CBEP rules and planning trace
schema/        canonical method catalogue and EvidenceResult schema
pipeline/      execution dispatch and evidence normalization
layers/        traditional ML and LLM evidence runners
report/        report model, ordering, provenance, HTML template, and PDF export
data/          five formal S5 cases and one S6 local validation case
outputs/       generated evidence artifacts and final reports
tests/         unit/regression suite and optional integration coverage
```

## Installation

UAGF-XAI supports Python 3.12. Install the base package with:

```bash
pip install uagf-xai
```

For PDF export, install the PDF extra and the Playwright Chromium runtime:

```bash
pip install "uagf-xai[pdf]"
python -m playwright install chromium
```

For local LLM execution support:

```bash
pip install "uagf-xai[llm]"
```

Install both optional capabilities with:

```bash
pip install "uagf-xai[llm,pdf]"
python -m playwright install chromium
```

Run the formal S5-driven workflow with externally supplied audit resources:

```bash
uagf-xai \
  --mode s5 \
  --s4-json path/to/s4.json \
  --s5-json path/to/s5.json
```

The S4/S5 JSON documents, datasets, and model artifacts are supplied by the
user; they are not included in the PyPI distribution. Relative resource paths
are resolved from the current working directory, and generated evidence and
reports are written under `./outputs/`. Use `--no-pdf` when PDF support is not
installed.

The thesis-specific local DistilGPT2 and MiniLM validation resources are not
included in the PyPI distribution. Local mode remains available only when the
caller supplies the required local validation resources.

Only load `.pickle`, `.pkl`, and `.joblib` model artifacts from trusted
sources. Deserializing an untrusted Python model artifact can execute arbitrary
code.

Minimal Python API usage:

```python
import json
from pathlib import Path

from uagf_xai import AuditAdapter, GovernanceAdapter, audit

s4_data = json.loads(Path("path/to/s4.json").read_text(encoding="utf-8"))
s5_data = json.loads(Path("path/to/s5.json").read_text(encoding="utf-8"))

governance_context = GovernanceAdapter.from_cgsa_report(s4_data)
audit_context = AuditAdapter.from_audit_report(s5_data)
results = audit(audit_context, governance_context, generate_pdf=False)
```

UAGF-XAI generates technical audit evidence. It does not make a final legal
compliance determination.

## Source-Checkout Development Environment

The frozen environment uses Python 3.12.10 and scikit-learn 1.8.0.

Activate the virtual environment venv312:
   ```bash
   # Windows (CMD/Powershell)
   venv312\Scripts\activate

   # Windows (git bash)
   source venv312/Scripts/activate

   # macOS/Linux
   source venv312/bin/activate
   ```

And then Install the dependencies:
   ```bash
   pip install -r requirements.txt
   ```

PDF export requires Playwright Chromium:

```bash
python -m playwright install chromium
```

## Running the Six Cases

Formal S5 cases:

```bash
python main.py --mode s5 --s4-json data/01_finclear_gmbh/s4_finclear-creditguard-001.json --s5-json data/01_finclear_gmbh/s5_finclear_gmbh_audit_state.json
python main.py --mode s5 --s4-json data/02_retailiq_ag/s4_retailiq-demandpulse-001.json --s5-json data/02_retailiq_ag/s5_retailiq_ag_audit_state.json
python main.py --mode s5 --s4-json data/03_harbourlogistik_gmbh/s4_harbourlogistik-harboursense-001.json --s5-json data/03_harbourlogistik_gmbh/s5_harbourlogistik_gmbh_audit_state.json
python main.py --mode s5 --s4-json data/04_legalmindd_ai_ltd/s4_legalmindd-lexai-001.json --s5-json data/04_legalmindd_ai_ltd/s5_legalmindd_ai_ltd_audit_state.json
python main.py --mode s5 --s4-json data/05_talentsift_gmbh/s4_talentsift-talentrank-001.json --s5-json data/05_talentsift_gmbh/s5_talentsift_gmbh_audit_state.json
```

S6 local LLM execution validation:

```bash
python main.py --mode local
```

All commands generate HTML and PDF by default. Use `--no-pdf` only for an
intentional HTML-only development run.

## Tests

Run the default offline suite:

```bash
python -m pytest -q
```

The final suite contains 18 test modules and 120 test functions. Parametrization
produces 134 passing cases in the default offline suite. It covers adapters,
resource contracts, exact-model integrity, CBEP, compatibility filtering,
evidence normalization, traditional and LLM methods, report semantics,
provenance, ordering, and PDF export.

One additional real-browser smoke test is optional integration coverage:

```bash
python -m pytest -m integration
```

## Interpretation Boundaries

- UAGF-XAI produces technical audit evidence; it does not issue a final legal
  compliance decision.
- Evidence completeness is scoped to the implemented catalogue and rules.
- CBEP minimum sufficiency is operational, not proof of globally minimal legal
  evidence.
- Fairness evidence uses only explicitly declared sensitive columns.
- Counterfactuals are mathematical explanations, not actionable
  recommendations.
- Drift measures input distributions, not label-based concept drift or proven
  performance degradation.
- MAPIE forecasting intervals do not model temporal dependence explicitly.
- LegalMind is metadata-only; local DistilGPT2 results must not be attributed
  to it.
- `minio://` URIs require a pre-populated local cache; authenticated MinIO
  download is not implemented.
