Metadata-Version: 2.4
Name: universal-test-framework
Version: 1.0.0
Summary: Universal polyglot test generation framework with 8-section contract enforcement
License-Expression: MIT
Project-URL: Homepage, https://github.com/phoenice-labs/Universal-Test-Framework
Project-URL: Repository, https://github.com/phoenice-labs/Universal-Test-Framework
Project-URL: Documentation, https://github.com/phoenice-labs/Universal-Test-Framework#readme
Project-URL: Bug Tracker, https://github.com/phoenice-labs/Universal-Test-Framework/issues
Project-URL: Changelog, https://github.com/phoenice-labs/Universal-Test-Framework/releases
Keywords: testing,mcp,test-generation,traceability,copilot,ai-testing,llm
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Typing :: Typed
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastmcp>=3.0.0
Requires-Dist: pyyaml==6.0.2
Requires-Dist: jinja2==3.1.4
Requires-Dist: pydantic>=2.11.0
Requires-Dist: httpx>=0.28.1
Requires-Dist: uvicorn>=0.29.0
Requires-Dist: starlette>=0.37.2
Requires-Dist: rich>=13.9.4
Requires-Dist: click>=8.1.7
Provides-Extra: dev
Requires-Dist: pytest==8.2.0; extra == "dev"
Requires-Dist: pytest-asyncio==0.23.6; extra == "dev"
Requires-Dist: pytest-cov==5.0.0; extra == "dev"
Requires-Dist: ruff==0.4.4; extra == "dev"
Requires-Dist: mypy==1.10.0; extra == "dev"
Requires-Dist: types-pyyaml==6.0.12.20240311; extra == "dev"
Dynamic: license-file

# Universal Test Framework (UTF)

> **Polyglot, requirements-driven test generation with hard contract enforcement — via MCP, VS Code, CLI, or Python SDK.**

UTF is an MCP server and test-generation engine that enforces an **8-section test contract** on every test it produces, regardless of language, project, or prompt quality. Tests that fail the contract are **blocked** — never silently included — so every test that reaches your suite is traceable, meaningful, and gap-honest.

[![PyPI version](https://img.shields.io/pypi/v/universal-test-framework)](https://pypi.org/project/universal-test-framework/)
[![Python](https://img.shields.io/pypi/pyversions/universal-test-framework)](https://pypi.org/project/universal-test-framework/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

---

## Key Features

- **12 MCP tools** — generate, validate, analyze, trace, execute, mutate, report, and query tests
- **6 languages** — Python, TypeScript, JavaScript, Java, Go, C++
- **7 test types** — unit, integration, API, E2E, contract, performance, security
- **8-section contract** — every test must pass all 8 sections or it is blocked (minimum score: 0.85)
- **Three install modes** — `uvx` (zero-clone), local venv, or Docker
- **VS Code @utf** — chat participant with slash commands
- **CI/CD ready** — GitHub Actions, GitLab CI, pre-commit hooks
- **Per-project overrides** — YAML rules in `.utf/rules/` override global defaults

---

## Quick Start

### Option 1 — Zero Install (uvx, recommended)

Requires [uv](https://docs.astral.sh/uv/getting-started/installation/). No cloning, no venv management.

Add to `%APPDATA%\Code\User\mcp.json` (Windows) or `~/.config/Code/User/mcp.json` (macOS/Linux):

```json
{
  "servers": {
    "utf": {
      "type": "stdio",
      "description": "Universal Test Framework — 8-section contract enforcement",
      "command": "uvx",
      "args": ["--from", "universal-test-framework", "utf-server", "--transport", "stdio"],
      "cwd": "${workspaceFolder}"
    }
  }
}
```

### Option 2 — Local Clone (VS Code + Copilot)

```powershell
git clone https://github.com/phoenice-labs/Universal-Test-Framework
cd Universal-Test-Framework
.\scripts\install-vscode-mcp.ps1   # registers MCP server, installs @utf extension, copies prompts
```

Reload VS Code → open Copilot Chat → type `@utf /generate-tests unit`.

### Option 3 — Docker (Team / Remote)

```bash
docker compose up -d utf-server
# MCP server available at http://localhost:8765/sse
```

---

## MCP Tools Reference

| Tool | Description |
|------|-------------|
| `generate_tests` | Generate a complete test suite satisfying the 8-section contract |
| `validate_test_contract` | Validate any test (generated or hand-written) against the contract |
| `analyze_coverage` | Identify coverage gaps in an existing test suite |
| `build_traceability_matrix` | Build a requirements → tests traceability matrix |
| `suggest_test_types` | Recommend test types with rationale from source code |
| `detect_language_framework` | Auto-detect programming language and test framework |
| `query_registry` | Query the persistent per-project test registry |
| `run_tests` | Execute test files and return structured CI results |
| `run_mutation_tests` | Run mutation testing and return mutation score |
| `feedback_status` | Get gap analysis, coverage health, and trend report |
| `generate_report` | Generate HTML / JUnit / JSON contract compliance report |
| `health` | Server health check: version, uptime, tool count |

---

## The 8-Section Test Contract

Every test generated by UTF must satisfy all 8 sections. Tests that fail any hard section are **blocked** — returned in `blocked_tests`, never silently included.

| # | Section | What It Must Contain | Weight |
|---|---------|----------------------|--------|
| 1 | `test_id` | Unique ID: `TC-{TYPE}-{NNN}` (e.g. `TC-US-042`) | 10% |
| 2 | `why_generated` | Rationale tied to a requirement (≥50 chars) | 10% |
| 3 | `requirement_mapping` | At least one `US-`, `AC-`, `REQ-`, `JIRA-`, `BUG-`, or `NFR-` reference | 15% |
| 4 | `how_it_exercises` | GIVEN / WHEN / THEN with inputs, mocks, assertions (≥100 chars) | 20% |
| 5 | `coverage_contribution` | Coverage type + module + estimated % | 15% |
| 6 | `expected_outcome` | Precise return values, status codes, state changes (≥50 chars) | 15% |
| 7 | `gaps_missing` | Honest list of what this test does NOT cover (≥40 chars) | 10% |
| 8 | `meaningfulness_check` | Self-assessment: meaningful / redundant / hallucinated (≥50 chars) | 5% |

**Minimum passing score: 0.85.** Tests below this threshold are blocked regardless of individual section presence. "none" in gaps or vague rationale like "to test the function" are rejected.

---

## Supported Matrix

| Language | Frameworks | Test Types |
|----------|------------|------------|
| Python | pytest | unit, integration, api, e2e, security, performance |
| TypeScript | Jest, Vitest, Playwright | unit, integration, e2e, api |
| JavaScript | Jest, Vitest | unit, integration |
| Java | JUnit 5 + AssertJ, Maven | unit, integration, api |
| Go | go-test + testify | unit, integration, api |
| C++ | Google Test | unit |

---

## VS Code Integration

After running `.\scripts\install-vscode-mcp.ps1` (Option 2) or adding the `uvx` MCP entry (Option 1):

### Copilot Chat Slash Commands

| Command | Effect |
|---------|--------|
| `@utf /generate-tests unit` | Unit tests for selected/active code |
| `@utf /generate-tests api` | API tests |
| `@utf /generate-tests integration` | Integration tests |
| `@utf /generate-tests e2e` | End-to-end tests |
| `@utf /generate-tests security` | Security / auth tests |
| `@utf /generate-tests performance` | Performance / load tests |
| `@utf /validate-contract` | Validate a test against the 8-section contract |
| `@utf /coverage-gaps` | Identify coverage gaps in the current suite |
| `@utf /traceability` | Build requirements → tests traceability matrix |
| `@utf /report` | Generate HTML contract compliance report |
| `@utf /status` | Server health and installation status |

### Minimal Prompt Example

You do not need to provide detailed requirements. UTF infers them:

```
@utf /generate-tests api
```

UTF reads the open file, detects language and framework, infers requirements from function signatures, generates happy-path + negative + edge-case tests — all validated against the 8-section contract.

---

## Python SDK

Use UTF directly in scripts or CI pipelines without the MCP server:

```python
from mcp_server.tools.generate_tests import generate_tests
from mcp_server.tools.validate_contract import validate_test_contract

# Generate tests — UTF infers language, framework, and requirements
result = generate_tests(
    test_type="unit",
    source_code=open("src/auth.py").read(),
    requirements_text="US-101: Users must be authenticated before accessing dashboard",
)

print(result["suite_code"])          # Executable test file
print(result["traceability_matrix"]) # Requirements → tests mapping
print(result["gaps"])                # Identified coverage gaps

# Validate any existing test
validation = validate_test_contract(test_content=my_test_markdown)
print(f"Score: {validation['score']:.0%}  Valid: {validation['is_valid']}")
```

---

## CI/CD Integration

### GitHub Actions

```yaml
# .github/workflows/test-quality-gate.yml
name: UTF Contract Gate
on: [push, pull_request]
jobs:
  contract-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install universal-test-framework
      - name: Validate test contract
        run: |
          python - <<'EOF'
          from mcp_server.tools.validate_contract import validate_test_contract
          import glob, sys, pathlib

          failures = []
          for f in glob.glob("tests/**/*.py", recursive=True):
              content = pathlib.Path(f).read_text()
              result = validate_test_contract(test_content=content)
              if not result["is_valid"]:
                  failures.append(f"{f}: score {result['score']:.0%}")
          if failures:
              print("Contract failures:\n" + "\n".join(failures))
              sys.exit(1)
          print("All tests passed contract validation")
          EOF
```

---

## Per-Project Configuration

Create `.utf/rules/project-overrides.yaml` in your project root:

```yaml
# Raise minimum score for safety-critical code
contract:
  min_score: 0.90          # default: 0.85
  hard_block_below: 0.75

# Match your Jira project key
traceability:
  requirement_id_pattern: "^(PROJ-\\d+|AC-\\d+(\\.\\d+)?|NFR-\\d+)$"

# Tests generated per requirement per type
scenario_counts:
  happy_path: 1
  negative: 2
  boundary: 1
  edge_case: 1

# Coverage advisory thresholds (appear in report, do not block)
coverage:
  line_target: 85
  branch_target: 75
  mutation_score_target: 70
```

Rules are deep-merged: project overrides layer on top of UTF's global defaults. The 8-section contract structure itself cannot be overridden.

---

## Architecture

```
UTF MCP Server (stdio or HTTP/SSE)
│
├── mcp_server/server.py         — FastMCP entrypoint, 12 registered tools
│
├── mcp_server/engine/           — Core processing
│   ├── language_detector.py     — Detects language + framework from code/path
│   ├── rule_engine.py           — Loads and merges YAML rules
│   ├── contract_validator.py    — Enforces 8-section contract, scores tests
│   ├── template_renderer.py     — Jinja2 test file generation
│   ├── context_resolver.py      — Resolves project context for generation
│   └── framework_mapper.py      — Maps language → framework → test runner
│
├── mcp_server/tools/            — One module per MCP tool
├── mcp_server/registry/         — SQLite per-project test registry
├── mcp_server/execution/        — pytest, Vitest, Maven, go-test adapters
├── mcp_server/mutation/         — mutmut, Stryker, PIT, Gremlins adapters
├── mcp_server/reporting/        — HTML, JUnit XML, JSON report generation
├── mcp_server/feedback/         — CI listener, trend analysis, gap reopener
│
├── rules/                       — Global YAML rules (language, framework, type)
├── templates/                   — Jinja2 test templates per language/framework
├── agent-customization/         — VS Code copilot-instructions + prompt palette
└── vscode-extension/            — @utf VS Code Chat Participant (VSIX)
```

**Transport modes:**
- `stdio` — default, used by VS Code MCP client and `uvx`
- `HTTP/SSE` — for remote/team deployment (`--transport http --port 8765`)

**SQLite registry** resolves to `.utf/utf.db` relative to `cwd` (the open workspace folder). Each project has its own isolated registry — no shared state.

---

## Installation Options Summary

| Mode | Command | Requirements |
|------|---------|--------------|
| **uvx** (zero-clone) | `uvx --from universal-test-framework utf-server` | [uv](https://docs.astral.sh/uv/) |
| **pip** | `pip install universal-test-framework` | Python 3.11+ |
| **Local clone** | `.\scripts\install-vscode-mcp.ps1` | Git, Python 3.11+ |
| **Docker** | `docker compose up -d utf-server` | Docker |

---

## Security

- All MCP tool inputs are validated via Pydantic before processing
- No secrets, credentials, or PII are logged or stored
- The registry (`utf.db`) is local to each project and never transmitted
- Docker image runs as non-root user
- Rate limiting is enforced on the HTTP/SSE transport

---

## License

MIT — see [LICENSE](LICENSE)
