Metadata-Version: 2.4
Name: trunft
Version: 0.6.0
Summary: TruNFT - unified Non-Functional Testing platform (CLI + MCP server) for Web, Mobile Native, Mobile Browser and Mobile Hybrid apps
Author-email: Shan Konduru <ShanKonduru@gmail.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/ShanKonduru/TruNFT
Project-URL: Source, https://github.com/ShanKonduru/TruNFT
Project-URL: Issues, https://github.com/ShanKonduru/TruNFT/issues
Keywords: non-functional-testing,performance-testing,load-testing,security-testing,chaos-engineering,accessibility,compatibility-testing,mobile-testing,k6,mcp
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: System :: Benchmark
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: PyYAML>=6.0
Provides-Extra: web
Requires-Dist: playwright>=1.45; extra == "web"
Provides-Extra: mobile
Requires-Dist: Appium-Python-Client>=4.0; extra == "mobile"
Provides-Extra: chaos
Requires-Dist: chaostoolkit>=1.19; extra == "chaos"
Provides-Extra: mcp
Requires-Dist: mcp<3,>=2.3; extra == "mcp"
Provides-Extra: all
Requires-Dist: trunft[chaos,mcp,mobile,web]; extra == "all"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: anyio>=4; extra == "dev"
Requires-Dist: mcp<3,>=2.3; extra == "dev"
Dynamic: license-file

# TruNFT — Unified Non-Functional Testing Platform

One framework, one CLI, one config and one report for **non-functional testing** across:

| Platform | Key | What is exercised |
|---|---|---|
| Web UI | `web` | Site URLs in desktop browsers |
| Mobile browser | `mobile-browser` | Site URLs with mobile form factor / device emulation |
| Mobile native | `mobile-native` | App binary (APK/IPA) + the backend APIs it calls |
| Mobile hybrid | `mobile-hybrid` | App binary + embedded web content + backend APIs |

TruNFT is an **orchestrator**. It sends each kind of test to an established tool (k6, OWASP ZAP, Toxiproxy, Chaos Toolkit, Lighthouse, Playwright, adb/xcrun), applies the same SLO thresholds to the results, and writes JSON, JUnit and HTML reports.

## Coverage

| Pillar (`--pillar`) | Test types (`-t`) | Engine |
|---|---|---|
| **Performance** `performance` | load, stress, spike, endurance (soak), scalability, capacity | k6 (one parameterised script, profiles in config) |
| **Security** `security` | vulnerability, penetration, access_control | OWASP ZAP baseline/full (Docker); built-in role × endpoint authZ matrix |
| **Reliability & availability** `reliability` | stability, failover, recovery, disaster_recovery | Background availability probe + configurable trigger/restore hooks (RTO, outage) |
| **Resilience & chaos** `chaos` | network_latency, service_failure, resource_exhaustion, experiment | Toxiproxy, shell hooks (kubectl / docker / AWS FIS), Chaos Toolkit |
| **Usability & accessibility** `accessibility` | accessibility, usability | Lighthouse (axe-core rules, Core Web Vitals); mobile vs desktop form factor |
| **Compatibility** `compatibility` | browser, device, os, configuration | Playwright (Chromium/Firefox/WebKit, device emulation, locale/dark mode/…) |
| **Installability & portability** `installability` | install, upgrade, uninstall, portability | adb (Android), xcrun simctl (iOS); multi-host health checks |
| **Compliance & localization** `compliance` | security_headers, privacy_cookies, localization | Built-in HTTP checks (OWASP headers, cookie flags, per-locale `lang` and untranslated strings) |

Run `trunft list` for the full platform × test-type matrix.

## Two interfaces, one engine

```
           trunft (CLI)          trunft-mcp (MCP server: stdio / streamable-http)
                 \                      /
                  +---- trunft.api ----+      capabilities, doctor, plan, run, reports
                            |
              runner -> pillars -> external tools (k6, ZAP, Lighthouse, ...)
                            |
               reports/<run>/results.json, junit.xml, report.html
```

Both interfaces read the same config and write the same reports, so a run started from one can be inspected from the other.

## Quick start (CLI)

```bash
python -m venv .venv && .venv/Scripts/activate     # source .venv/bin/activate on Linux/macOS
pip install -e ".[dev]"            # add [web] for Playwright, [mobile] for Appium client
trunft doctor                      # shows which external tools are available
trunft setup                       # installs missing ones (winget/brew/npm/pip); --dry-run to preview
trunft plan -e staging --pillar performance       # preview, runs nothing
trunft run --dry-run               # plan everything, execute nothing
trunft run -e dev -p web --pillar compliance
trunft run -e staging --pillar performance -t load -t spike
trunft reports                     # recent runs
trunft report <run> --failures     # what broke and which thresholds were breached
pytest                             # framework self-tests
```

`list`, `doctor`, `plan`, `reports` and `report` accept `--json`.
Exit code: `0` = no failures, `1` = at least one FAILED/ERROR, `2` = bad config or arguments.

## MCP server (AI agents / IDEs)

```bash
pip install -e ".[mcp]"
trunft-mcp --root C:/myprojects/TruNFT                       # stdio
trunft-mcp --transport streamable-http --port 8765           # http://127.0.0.1:8765/mcp
trunft mcp                                                   # same server via the main CLI
```

Register it with Claude Code:

```bash
claude mcp add trunft --env TRUNFT_ROOT=C:/myprojects/TruNFT -- C:/myprojects/TruNFT/.venv/Scripts/trunft-mcp.exe
```

Or in Claude Desktop / any MCP client config:

```json
{ "mcpServers": { "trunft": {
    "command": "C:/myprojects/TruNFT/.venv/Scripts/trunft-mcp.exe",
    "env": { "TRUNFT_ROOT": "C:/myprojects/TruNFT" } } } }
```

| Tool | Purpose |
|---|---|
| `list_capabilities` | Pillars, test types, required tools, and which platforms each applies to |
| `check_tools` | Which external tools are installed |
| `list_environments` / `get_config` | Environments and the merged config (secrets redacted) |
| `plan_tests` | Preview the platform × pillar × test-type combinations |
| `start_test_run` | Start a background run and return a `run_id` at once. **`dry_run` defaults to `true`** |
| `get_test_run` / `cancel_test_run` | Poll progress and results, or stop the remaining tests |
| `list_test_reports` / `get_test_report` | Past runs from CLI or MCP; `only_failures` filter |

Runs are asynchronous because soak tests can last for hours. The agent starts a run, polls it, then reads the report. Tools that only read are marked read-only. `start_test_run` is marked destructive and open-world because real runs generate load, run security scans and can inject faults.

### Result statuses
| Status | Meaning |
|---|---|
| `passed` | Ran, and every threshold was met |
| `failed` | Ran, but breached a threshold or assertion |
| `error` | Couldn't complete (tool crashed, target unreachable) |
| `skipped` | Dry-run, not applicable to the platform, or a required tool isn't installed |
| `not_implemented` | Placeholder: needs config (e.g. trigger hook) or code; the message says what |

## Layout

```
config/trunft.yaml              master config: targets, thresholds (SLOs), pillar settings
config/environments/*.yaml      per-environment overrides (dev, staging, ...); ${ENV_VAR:-default} supported
src/trunft/api.py               service layer shared by CLI and MCP server
src/trunft/cli.py               `trunft` command line
src/trunft/mcp_server.py        `trunft-mcp` MCP server (stdio / streamable-http)
src/trunft/core/                config, models, pillar base + registry, runner, thresholds, probes
src/trunft/pillars/*.py         one module per NFT pillar (auto-discovered)
src/trunft/reporting/           results.json, junit.xml, report.html
workloads/k6/scenario.js        parameterised k6 workload - put real user journeys here
chaos/experiments/              Chaos Toolkit experiments (*.json)
docker/docker-compose.yml       toxiproxy, influxdb, grafana for local runs
apps/android, apps/ios          app binaries under test (git-ignored)
.github/workflows/ci.yml        unit tests on push/PR (Linux + Windows); no live tests in CI
.github/workflows/publish.yml   gated PyPI release on v* tags
```

## Thresholds (SLO gates)

Every pillar returns metrics. A threshold `max_<metric>` or `min_<metric>` under `thresholds.<pillar>.default`, or under a per-test-type block, decides pass or fail:

```yaml
thresholds:
  performance:
    default: {max_p95_ms: 800, max_error_rate: 0.01}
    spike:   {max_p95_ms: 1500}
```

## Step-load analysis (knee and breaking point)

k6 summarises a whole test as one set of numbers, so a ramp from 25 to 400 users averages very different load levels together. When a performance profile has stages, TruNFT also records k6's time-stamped samples and cuts the run into **load steps**, measuring users, req/s, p50/p95/p99 and error rate for each one.

Use the `staircase` shape to hold each level steady, so every step is measured under stable load:

```yaml
pillars:
  performance:
    profiles:
      stress:
        executor: ramping-vus                 # or ramping-arrival-rate (levels = iterations/s)
        staircase: {levels: [50, 100, 200, 400, 800], ramp: 30s, hold: 3m}
        step_slo: {p95_ms: 800, error_rate: 0.01}   # judges each step
```

| Metric | Meaning |
|---|---|
| `max_users_within_slo` | Users at the last step of the opening run of steps that stayed within the step SLO |
| `sustained_rps` | Highest req/s held across a full step while within the step SLO |
| `saturation_users` | The knee: the last step before throughput stopped scaling (the next step added load but gained less than half as much in req/s) |
| `breaking_point_users` | Users at the first step whose error rate breached the step SLO |
| `load_steps` | Number of steps measured |

- **Gating.** These metrics are gated like any other (`min_max_users_within_slo`, `min_sustained_rps`, ...). Stress tests overload the app on purpose, so the bundled config judges them on these metrics rather than on whole-run latency.
- **Capacity.** For `capacity`, `rps` is the sustained rate, and the whole-run average is kept as `avg_rps`.
- **Step SLO.** Defaults to the test type's `max_p95_ms` / `max_error_rate`, then to the profile's `abort_on`.
- **Plain ramps.** A profile with no steady holds is cut into fixed windows instead (`pillars.performance.steps: {window_s: 15}`).
- **Report and files.** The HTML report shows a step table plus throughput-vs-users and p95-vs-users charts with the limit and the knee marked. Raw data goes to `steps.json` and `k6-samples.csv.gz`.

## Benchmark comparison (▲ better / ▼ worse)

Absolute limits miss regressions that stay inside the limit: a p95 of 140 ms passes `max_p95_ms: 150` even if the last release managed 20 ms. Store a known-good run as the **benchmark** and every later run is compared with it:

```bash
trunft baseline set 20261105-213636-sut       # the run's passed tests become the benchmark for its environment
trunft baseline set --latest-passing -e sut   # or, per test, the newest run where it passed
trunft baseline show -e sut                   # what is stored  (clear [--test performance/load] removes)
trunft compare <runA> <runB>                  # any two runs, B judged against A (--all lists unchanged metrics)
```

**Verdicts.** Each metric gets **▲ improvement**, **▼ regression** or **≈ no change**. Changes within `baseline.noise_pct` (default ±5 %) count as no change.

**Direction.** Whether a rise is good depends on the metric:

| Better when lower | Better when higher |
|---|---|
| latency, errors, memory, CPU, outage/recovery time, alerts, violations, failures | `rps`, `sustained_rps`, availability, scores, `max_users_within_slo` |

Counts such as `requests` are shown but not judged.

**Regression gates.** `max_<metric>_regression_pct` fails a test that got that much worse than its benchmark, even while it is still within its absolute limit:

```yaml
thresholds:
  performance:
    load: {max_p95_ms: 150, max_p95_ms_regression_pct: 100, max_rps_regression_pct: 25}
```

Metrics whose benchmark value is 0 get a verdict but no percentage, so the gates skip them; absolute limits cover those.

**Like-for-like warnings.** Each result records a fingerprint: TruNFT version, target URL, a hash of the test's configuration (such as the workload profile) and the project's git commit. A comparison warns when the target or configuration differs from the benchmark's.

**In the report:** a "vs benchmark" column in every metric table, a ▲/▼ summary tile and a "Regressions vs benchmark" list.

**Storage.** Benchmarks live in `reports/baselines/<env>.json`.

**MCP tools:** `set_baseline`, `get_baseline` and `compare_runs`.

## Execution history, trends and heatmap

Every run that isn't a dry run is recorded in `reports/history.db` (SQLite), so history survives clean-ups of the run folders.

```bash
trunft history import            # once: backfill from existing reports/**/results.json
trunft history -e sut --open     # build reports/history.html and open it
trunft history show -e sut       # text summary: pass rate per run, newly failing, flaky
```

`history.html` is a single self-contained file. It contains:

- **Pass rate per run** over time. Click a point to open that run's report.
- **A heatmap** of tests × runs, coloured by status with a ✓/✗ glyph. A row that alternates colours points to a flaky test.
- **Newly failing tests:** failing now but passed before, with the run where the failures started and the last passing run.
- **Flaky tests:** outcome changed 2+ times in their last 10 runs.
- **Metric trends** per test: p95, req/s, sustained req/s, error rate, users within SLO, memory growth, availability and more. Each chart draws the **limit** and the **benchmark** as reference lines.
- **Filters** for environment, label (for example `TruShop (healthy)` vs `(flawed)`), platform, test area and number of runs.

**Settings:** `history: {db: reports/history.db, keep_runs: 200, enabled: true}`.

**MCP tools:** `get_history` and `get_trend(test, metric)`.

**TruShop scripts:** `run-trushop-tests` refreshes the dashboard after every run; `--history` opens it.

## Try it against the bundled test app (TruShop)

`sut/` contains a Dockerised system under test that exercises every test type: a load-balanced web tier with a DR replica, a catalog microservice behind Toxiproxy, i18n pages, auth roles and fault-injection hooks. It can also switch on deliberate defects so you can see what each check catches.

```bash
run-trushop-tests.bat        # Windows: guided menus, starts TruShop, opens the HTML report
./run-trushop-tests.sh       # macOS / Linux
```

Or by hand:

```bash
docker compose -f sut/docker-compose.yml up -d --build
trunft run -e sut -p web --pillar compliance --pillar reliability --pillar chaos
```

See [sut/README.md](sut/README.md).

## Installing from PyPI

```bash
pip install trunft            # core; extras: trunft[web], trunft[mcp], trunft[chaos], trunft[all]
trunft init my-nft-project    # creates config/, workloads/k6, chaos/, docker/ from bundled templates
cd my-nft-project && trunft setup && trunft run --dry-run
```

## Server-side resource monitoring

Latency and error rates are measured from the client, so they can't see a memory leak until it takes the service down. To catch one sooner, add a `monitoring` block. TruNFT then samples container memory and CPU throughout each monitored test:

```yaml
monitoring:
  pillars: [performance]
  docker: {containers: [my-api-1, my-api-2], interval_s: 5, warmup_fraction: 0.2}
```

| Metric | Meaning | Default gate |
|---|---|---|
| `mem_growth_mb_per_min` | Least-squares slope of memory after warm-up: the leak indicator | endurance: ≤ 5 |
| `mem_growth_mb` | End memory minus start memory | none |
| `mem_peak_pct` | Highest memory use as a % of the container limit | ≤ 90 |
| `cpu_peak_pct` | Highest CPU use (100 = one core) | none |

The values are the worst case across containers. Per-container detail goes to `resources.json` next to each result. Only Docker is supported so far; the sampler is designed so that `kubectl top` or a cloud metrics source can be added later.

## Releasing

Publishing uses PyPI **Trusted Publishing** (`.github/workflows/publish.yml`, no API tokens):

1. Bump `__version__` in `src/trunft/__init__.py` (the single source of the version) and commit.
2. Tag that commit and push the tag: `git tag v0.2.0 && git push origin v0.2.0`

The workflow uploads only if **all** of these gates pass:

| Gate | Check |
|---|---|
| Version | Tag `v<x.y.z>` == `__version__` == built sdist/wheel versions |
| Unit | `pytest` on Linux + Windows, Python 3.10 and 3.13 |
| Security | Sources compile; Bandit SAST with zero findings; `pip-audit --strict` on the runtime dependency set |
| Build | `twine check --strict`; the wheel is installed into a clean venv, then `trunft init` and a dry run are executed |

Then it publishes to PyPI (GitHub environment `pypi`).
Run the version gate locally with `python scripts/check_release_version.py v0.2.0`.

If you edit files under `config/`, `workloads/`, `chaos/` or `docker/`, copy them to `src/trunft/templates/project/`. A test fails if the two copies drift apart.

## Extending

Add a module under `src/trunft/pillars/`:

```python
from trunft.core.pillar import Pillar, RunContext, register
from trunft.core.models import Status

@register
class Energy(Pillar):
    name, title = "energy", "Battery & Energy"
    test_types = {"drain": "Battery drain over a 30-min journey"}
    type_platforms = {"drain": Platform.mobile_apps()}
    tools = {"drain": ["adb"]}

    def run(self, ctx: RunContext, test_type: str):
        ...
        return self.result(ctx, test_type, Status.PASSED, metrics={"mah_per_hour": 120})
```

## Next up

All three agreed [BACKLOG.md](BACKLOG.md) items (step-load analysis, benchmark comparison, execution history) are done. Still open: a shared history store so CI and laptops see the same history, plus the smaller items listed at the end of BACKLOG.md.

## Roadmap: open placeholders (`not_implemented`)

- **Mobile client performance**: startup time, memory, CPU and frame rate (adb `dumpsys`, Xcode Instruments)
- **Mobile security**: MobSF static/dynamic analysis of APK/IPA
- **Native accessibility**: Espresso Accessibility Test Framework, XCUITest `performAccessibilityAudit`
- **Device-cloud compatibility**: Appium or Selenium grid (BrowserStack, Sauce Labs, LambdaTest) for the OS and native matrix
- **Upgrade data retention**, **DR RPO verification**, **client-side network conditioning** (offline/3G)
- **Native localization**: resource-bundle key completeness plus per-locale screenshots
- **Compliance evidence packs**: GDPR, PCI-DSS, HIPAA, WCAG conformance statement

> ⚠️ Run active security scans (`penetration`) and chaos/DR hooks only against environments you are authorised to test. `allow_active_scan` is `false` by default.
